rustyrazorblade commented on issue #110:
URL:
https://github.com/apache/cassandra-easy-stress/issues/110#issuecomment-6347675402
## Cluster test results for #111
I ran 100 short tests of the populate changes in PR #111 on an easy-db-lab
cluster. **100 of 100 pass.**
**Setup:**
- Cassandra 5.0.9 on 3 × `i4i.xlarge`.
- Keyspaces with `SimpleStrategy` RF 3.
- Stress on 1 × `c6id.2xlarge`, with the image built from `182ee5f`.
**How each test works:**
- Each test uses its own keyspace and a 1-minute main phase.
- Writes use `LOCAL_QUORUM`.
- Unless a test says otherwise, the main phase is reads only (`-r 1`). So
the table after the run holds only what populate wrote.
- Each test checks the populate summary line (rows written, rows failed).
- Where the expected result is exact, the test also counts the table with
`cqlsh` at `QUORUM`. For counters, it checks `sum(value)`.
**Coverage:**
- `--populate N`, `10k`, `all`, `-1`, and `0`.
- `--populate.concurrency` from 1 to 2000.
- `--populate.generator` set to `sequence`, `random`, or `normal`, and the
workload override.
- `--populate.batch` from 1 to 1000, with `UNLOGGED` and `COUNTER` batches.
- `--populate.deletes` and `--populate.deleteratio`.
- 1 to 8 threads.
- Interaction with `--rate`, `--deleterate`, `--maxwlat`, and `--cl`.
- 12 workloads.
- Node failures:
- One node down before populate.
- One node stopped while populate runs.
- `--cl ALL` with a node down.
### Results that matter
- **No lost rows.** Every test with an exact expected count matched it.
This includes 400,000-row populates while `db1` was stopped part way through
(`pt084` to `pt086`, stopped 32 to 37 s in, populate 68 to 161 s).
- **No double counts.** The counter sum was exactly 400,000 in the
node-stop test, so no retry counted an increment twice.
- **Failures are counted, not hidden.** With `db2` down and `--cl ALL`,
populate reported all 2,000 rows as failed (`pt083`).
- **`--rate` does not slow populate.** With `--rate 50`, populate wrote
20,000 rows in 2 s (`pt022`).
- **`--deleterate` does not reach populate.** With `--deleterate 0.5` in
the main phase, populate printed `delete ratio: 0.0` (`pt021`).
### Problems found
1. **`--populate -1` wrote 1 row on the command line.**
`HumanReadableConverter` drops the sign. This is fixed in `182ee5f` on the PR,
and `pt003` checks it.
2. **`CreateDrop` fails every write, in the main phase too.** It calls
`bound.setString(...)` and discards the result, but driver 4 statements are
immutable. The bug was already there; the old populate hid it with
`resetErrors()`. Filed as #112.
3. **`Sets` fails every write** (#105, already open). Populate now reports
this: "0 rows written, 10000 rows failed".
4. **Populate failures have no cause in the console.** `PopulateLoop` logs
the cause with `log.error`, and that goes only to `stress.log`. The console
shows the count of failed rows but not the reason.
### Changes to the test plan during the run
I changed some expectations after the results. This list states each change:
- **`pt040`:** I expected a 1,000-row batch to go over
`batch_size_fail_threshold`. Cassandra skips that check for a batch with one
mutation (CASSANDRA-10876, `BatchStatement.verifyBatchSize`). The test now
expects all rows to be written, and they were.
- **`pt062`–`pt064` (`Sets`) and `pt081` (`CreateDrop`):** these now expect
every write to fail, because of problems 2 and 3. What they check is that
populate reports the failures.
- **Runner bugs, now fixed:**
- The first runner passed some options twice, which JCommander rejects.
`pt021`, `pt023`, `pt025`, `pt026`, `pt081`, and `pt083` were run again.
- In the first two attempts, the node-stop tests stopped `db1` after
populate had finished, so they did not test a failure. The runner now checks
this and fails the test if populate was no longer running. The reported
results are from the valid run.
- **`pt062`:** the first re-run failed to connect at startup, before
populate began. It was run again.
<details>
<summary>All 100 tests</summary>
| # | Workload | Test | Flags | Written | Failed | Checked | Result |
|---|---|---|---|---:|---:|---|---|
| 1 | KeyValue | row count, defaults | `--populate` 10000 `--partitions`
10000 | 10,000 | 0 | count 10,000 | ✅ |
| 2 | KeyValue | --populate all | `--populate` all `--partitions` 10000 |
10,000 | 0 | count 10,000 | ✅ |
| 3 | KeyValue | --populate -1 (same as all) | `--populate` `-1`
`--partitions` 10000 | 10,000 | 0 | count 10,000 | ✅ |
| 4 | KeyValue | human readable --populate 10k | `--populate` 10k
`--partitions` 20000 | 10,000 | 0 | count 10,000 | ✅ |
| 5 | KeyValue | all, 4 threads | `--populate` all `--partitions` 5000
`--threads` 4 | 20,000 | 0 | count 20,000 | ✅ |
| 6 | KeyValue | all, 8 threads | `--populate` all `--partitions` 2000
`--threads` 8 | 16,000 | 0 | count 16,000 | ✅ |
| 7 | KeyValue | populate > partitions wraps | `--populate` 20000
`--partitions` 5000 `--threads` 2 | 40,000 | 0 | count 10,000 | ✅ |
| 8 | KeyValue | concurrency 1 | `--populate` all `--partitions` 10000
`--threads` 2 `--populate.concurrency` 1 | 20,000 | 0 | count 20,000 | ✅ |
| 9 | KeyValue | concurrency 10 | `--populate` all `--partitions` 10000
`--threads` 2 `--populate.concurrency` 10 | 20,000 | 0 | count 20,000 | ✅ |
| 10 | KeyValue | concurrency 100 | `--populate` all `--partitions` 10000
`--threads` 2 `--populate.concurrency` 100 | 20,000 | 0 | count 20,000 | ✅ |
| 11 | KeyValue | concurrency 500 | `--populate` all `--partitions` 10000
`--threads` 2 `--populate.concurrency` 500 | 20,000 | 0 | count 20,000 | ✅ |
| 12 | KeyValue | concurrency 2000 | `--populate` all `--partitions` 10000
`--threads` 2 `--populate.concurrency` 2000 | 20,000 | 0 | count 20,000 | ✅ |
| 13 | KeyValue | generator sequence (explicit) | `--populate` 10000
`--partitions` 10000 `--populate.generator` sequence | 10,000 | 0 | count
10,000 | ✅ |
| 14 | KeyValue | generator random: collisions expected | `--populate` 10000
`--partitions` 10000 `--populate.generator` random | 10,000 | 0 | count 6,258 |
✅ |
| 15 | KeyValue | generator normal: collisions expected | `--populate` 10000
`--partitions` 10000 `--populate.generator` normal | 10,000 | 0 | count 4,327 |
✅ |
| 16 | KeyValue | batch 2 overwrites one row per partition | `--populate`
all `--partitions` 5000 `--populate.batch` 2 | 10,000 | 0 | count 5,000 | ✅ |
| 17 | KeyValue | batch 10 overwrites one row per partition | `--populate`
all `--partitions` 2000 `--populate.batch` 10 | 20,000 | 0 | count 2,000 | ✅ |
| 18 | KeyValue | --populate.deletes 0.1 | `--populate` all `--partitions`
10000 `--populate.deletes` 0.1 | 10,000 | 0 | count 9,054 | ✅ |
| 19 | KeyValue | --populate.deleteratio 0.5 (alias) | `--populate` all
`--partitions` 10000 `--populate.deleteratio` 0.5 | 10,000 | 0 | count 5,035 |
✅ |
| 20 | KeyValue | --populate.deletes 1.0 | `--populate` all `--partitions`
5000 `--populate.deletes` 1.0 | 5,000 | 0 | count 0 | ✅ |
| 21 | KeyValue | main --deleterate is not used by populate | `--populate`
all `--partitions` 10000 `-r` 0.5 `--deleterate` 0.5 | 10,000 | 0 | log only |
✅ |
| 22 | KeyValue | --rate 50 does not slow populate | `--populate` all
`--partitions` 20000 `--rate` 50 | 20,000 | 0 | count 20,000 | ✅ |
| 23 | KeyValue | optimizer (--maxwlat) starts after populate | `--populate`
all `--partitions` 10000 `--maxwlat` 20 `-r` 0.0 | 10,000 | 0 | log only | ✅ |
| 24 | KeyValue | --populate 0 skips populate | `--populate` 0
`--partitions` 10000 | | | count 0 | ✅ |
| 25 | KeyValue | --cl ONE | `--populate` all `--partitions` 10000 `--cl`
ONE | 10,000 | 0 | count 10,000 | ✅ |
| 26 | KeyValue | --cl ALL | `--populate` all `--partitions` 10000 `--cl`
ALL | 10,000 | 0 | count 10,000 | ✅ |
| 27 | KeyValue | large: 8 threads x 25k, concurrency 1000 | `--populate`
all `--partitions` 25000 `--threads` 8 `--populate.concurrency` 1000 | 200,000
| 0 | count 200,000 | ✅ |
| 28 | BasicTimeSeries | batch 1 | `--populate` 10000 `--partitions` 100
`--populate.batch` 1 | 10,000 | 0 | count 10,000 | ✅ |
| 29 | BasicTimeSeries | batch 5 | `--populate` 10000 `--partitions` 100
`--populate.batch` 5 | 10,000 | 0 | count 10,000 | ✅ |
| 30 | BasicTimeSeries | batch 25 | `--populate` 10000 `--partitions` 100
`--populate.batch` 25 | 10,000 | 0 | count 10,000 | ✅ |
| 31 | BasicTimeSeries | batch 100 | `--populate` 10000 `--partitions` 100
`--populate.batch` 100 | 10,000 | 0 | count 10,000 | ✅ |
| 32 | BasicTimeSeries | batch 50, 4 threads | `--populate` 5000
`--partitions` 100 `--populate.batch` 50 `--threads` 4 | 20,000 | 0 | count
20,000 | ✅ |
| 33 | BasicTimeSeries | all with batch 10 | `--populate` all `--partitions`
1000 `--populate.batch` 10 | 10,000 | 0 | count 10,000 | ✅ |
| 34 | BasicTimeSeries | generator random, batch 10 | `--populate` 10000
`--partitions` 1000 `--populate.batch` 10 `--populate.generator` random |
10,000 | 0 | count 10,000 | ✅ |
| 35 | BasicTimeSeries | generator normal, batch 5 | `--populate` 10000
`--partitions` 1000 `--populate.batch` 5 `--populate.generator` normal | 10,000
| 0 | count 10,000 | ✅ |
| 36 | BasicTimeSeries | deletes 0.2 with batch 10 (range deletes) |
`--populate` 10000 `--partitions` 100 `--populate.batch` 10
`--populate.deletes` 0.2 | 10,000 | 0 | count 9,790 | ✅ |
| 37 | BasicTimeSeries | concurrency 1, batch 20 | `--populate` 10000
`--partitions` 100 `--populate.batch` 20 `--populate.concurrency` 1 | 10,000 |
0 | count 10,000 | ✅ |
| 38 | BasicTimeSeries | concurrency 2000, batch 5, 4 threads | `--populate`
10000 `--partitions` 100 `--populate.batch` 5 `--populate.concurrency` 2000
`--threads` 4 | 40,000 | 0 | count 40,000 | ✅ |
| 39 | BasicTimeSeries | batch 200 | `--populate` 10000 `--partitions` 50
`--populate.batch` 200 | 10,000 | 0 | count 10,000 | ✅ |
| 40 | BasicTimeSeries | batch 1000 in one partition (Cassandra skips the
batch size check for one partition) | `--populate` 10000 `--partitions` 10
`--populate.batch` 1000 | 10,000 | 0 | count 10,000 | ✅ |
| 41 | BasicTimeSeries | large: 8 threads x 25k, batch 50 | `--populate`
25000 `--partitions` 200 `--populate.batch` 50 `--threads` 8
`--populate.concurrency` 500 | 200,000 | 0 | count 200,000 | ✅ |
| 42 | CountersWide | counter batch 1 | `--populate` 10000 `--partitions`
100 `--populate.batch` 1 | 10,000 | 0 | sum 10,000 | ✅ |
| 43 | CountersWide | counter batch 4 | `--populate` 10000 `--partitions`
100 `--populate.batch` 4 | 10,000 | 0 | sum 10,000 | ✅ |
| 44 | CountersWide | counter batch 20 | `--populate` 10000 `--partitions`
100 `--populate.batch` 20 | 10,000 | 0 | sum 10,000 | ✅ |
| 45 | CountersWide | counter batch 100 | `--populate` 10000 `--partitions`
100 `--populate.batch` 100 | 10,000 | 0 | sum 10,000 | ✅ |
| 46 | CountersWide | counter batch 10, 4 threads | `--populate` 5000
`--partitions` 100 `--populate.batch` 10 `--threads` 4 | 20,000 | 0 | sum
20,000 | ✅ |
| 47 | CountersWide | concurrency 1, batch 10 | `--populate` 5000
`--partitions` 100 `--populate.batch` 10 `--populate.concurrency` 1 | 5,000 | 0
| sum 5,000 | ✅ |
| 48 | CountersWide | concurrency 2000, batch 5 | `--populate` 10000
`--partitions` 100 `--populate.batch` 5 `--populate.concurrency` 2000 | 10,000
| 0 | sum 10,000 | ✅ |
| 49 | CountersWide | counter deletes 0.1 go alone | `--populate` 10000
`--partitions` 100 `--populate.batch` 10 `--populate.deletes` 0.1 | 10,000 | 0
| sum 9,891 | ✅ |
| 50 | CountersWide | generator random, batch 10 | `--populate` 10000
`--partitions` 100 `--populate.batch` 10 `--populate.generator` random | 10,000
| 0 | sum 10,000 | ✅ |
| 51 | CountersWide | all with batch 20 | `--populate` all `--partitions`
500 `--populate.batch` 20 | 10,000 | 0 | sum 10,000 | ✅ |
| 52 | Locking | 1 thread | `--partitions` 10000 | 10,000 | 0 | count
10,000, key present | ✅ |
| 53 | Locking | 4 threads use the per-thread prefix | `--partitions` 5000
`--threads` 4 | 20,000 | 0 | count 20,000, key present | ✅ |
| 54 | Locking | --populate is ignored by the workload option | `--populate`
5 `--partitions` 2000 | 2,000 | 0 | count 2,000 | ✅ |
| 55 | Locking | --populate.generator random wins over the workload |
`--partitions` 5000 `--populate.generator` random | 5,000 | 0 | count 3,146 | ✅
|
| 56 | Locking | --populate.deletes wins over the workload | `--partitions`
5000 `--populate.deletes` 0.2 | 5,000 | 0 | log only | ✅ |
| 57 | Locking | concurrency 1 | `--partitions` 2000
`--populate.concurrency` 1 | 2,000 | 0 | count 2,000 | ✅ |
| 58 | Locking | batch 5 | `--partitions` 2000 `--populate.batch` 5 | 2,000
| 0 | count 400 | ✅ |
| 59 | Maps | all | `--populate` all `--partitions` 10000 | 10,000 | 0 |
count 10,000 | ✅ |
| 60 | Maps | batch 5 merges map entries | `--populate` all `--partitions`
2000 `--populate.batch` 5 | 10,000 | 0 | count 2,000 | ✅ |
| 61 | Maps | 4 threads, concurrency 500 | `--populate` all `--partitions`
5000 `--threads` 4 `--populate.concurrency` 500 | 20,000 | 0 | count 20,000 | ✅
|
| 62 | Sets | all (known bug #105: every write fails; populate must report
it) | `--populate` all `--partitions` 10000 | 0 | 10,000 | count 0 | ✅ |
| 63 | Sets | batch 5 (known bug #105: every write fails; populate must
report it) | `--populate` all `--partitions` 2000 `--populate.batch` 5 | 0 |
10,000 | count 0 | ✅ |
| 64 | Sets | deletes 0.1 (known bug #105: inserts fail, deletes succeed) |
`--populate` all `--partitions` 10000 `--populate.deletes` 0.1 | 0 | 10,000 |
log only | ✅ |
| 65 | UdtTimeSeries | batch 1 | `--populate` 10000 `--partitions` 100 |
10,000 | 0 | count 10,000 | ✅ |
| 66 | UdtTimeSeries | batch 10 | `--populate` 10000 `--partitions` 100
`--populate.batch` 10 | 10,000 | 0 | count 10,000 | ✅ |
| 67 | UdtTimeSeries | batch 5, 4 threads | `--populate` 5000 `--partitions`
100 `--populate.batch` 5 `--threads` 4 | 20,000 | 0 | count 20,000 | ✅ |
| 68 | LWT | all | `--populate` all `--partitions` 5000 | 5,000 | 0 | count
5,000 | ✅ |
| 69 | LWT | batch 5 (conditional batch) | `--populate` all `--partitions`
1000 `--populate.batch` 5 | 5,000 | 0 | count 1,000 | ✅ |
| 70 | LWT | conditional deletes 0.1 go alone | `--populate` all
`--partitions` 5000 `--populate.deletes` 0.1 | 5,000 | 0 | log only | ✅ |
| 71 | LWT | 4 threads | `--populate` all `--partitions` 2000 `--threads` 4
| 8,000 | 0 | count 8,000 | ✅ |
| 72 | RandomPartitionAccess | batch 1 | `--populate` 10000 `--partitions`
1000 | 10,000 | 0 | count 9,595 | ✅ |
| 73 | RandomPartitionAccess | batch 10 | `--populate` 10000 `--partitions`
1000 `--populate.batch` 10 | 10,000 | 0 | count 9,551 | ✅ |
| 74 | RandomPartitionAccess | 4 threads, deletes 0.1 | `--populate` 5000
`--partitions` 1000 `--threads` 4 `--populate.deletes` 0.1 | 20,000 | 0 | log
only | ✅ |
| 75 | AllowFiltering | batch 1 | `--populate` 10000 `--partitions` 1000 |
10,000 | 0 | count 9,563 | ✅ |
| 76 | AllowFiltering | batch 10 | `--populate` 10000 `--partitions` 1000
`--populate.batch` 10 | 10,000 | 0 | count 9,549 | ✅ |
| 77 | AllowFiltering | 4 threads, deletes 0.1 | `--populate` 5000
`--partitions` 1000 `--threads` 4 `--populate.deletes` 0.1 | 20,000 | 0 | log
only | ✅ |
| 78 | SAI | batch 1 | `--populate` 10000 `--partitions` 1000 | 10,000 | 0 |
count 9,993 | ✅ |
| 79 | SAI | batch 10 | `--populate` 10000 `--partitions` 1000
`--populate.batch` 10 | 10,000 | 0 | count 9,994 | ✅ |
| 80 | SAI | 4 threads, deletes 0.1 | `--populate` 5000 `--partitions` 1000
`--threads` 4 `--populate.deletes` 0.1 | 20,000 | 0 | log only | ✅ |
| 81 | CreateDrop | existing CreateDrop bug: setString result is discarded,
so every insert fails; populate must report it | `--populate` 1000
`--partitions` 1000 `-r` 0.0 | 0 | 1,000 | log only | ✅ |
| 82 | KeyValue | db2 down before populate, LOCAL_QUORUM | `--populate` all
`--partitions` 20000 `--threads` 2 | 40,000 | 0 | count 40,000 | ✅ |
| 83 | KeyValue | db2 down before populate, --cl ALL: every row fails and is
counted | `--populate` all `--partitions` 2000 `--cl` ALL | 0 | 2,000 | log
only | ✅ |
| 84 | KeyValue | db1 stopped during populate | `--populate` all
`--partitions` 200000 `--threads` 2 `--populate.concurrency` 2 | 400,000 | 0 |
count 400,000 | ✅ |
| 85 | BasicTimeSeries | db1 stopped during populate, batch 5 | `--populate`
200000 `--partitions` 500 `--threads` 2 `--populate.batch` 5
`--populate.concurrency` 1 | 400,000 | 0 | count 400,000 | ✅ |
| 86 | CountersWide | db1 stopped during populate: retried counters may
count twice | `--populate` 200000 `--partitions` 200 `--threads` 2
`--populate.batch` 5 `--populate.concurrency` 1 | 400,000 | 0 | sum 400,000 | ✅
|
| 87 | KeyValue | repeat 1 | `--populate` all `--partitions` 10000
`--threads` 4 `--populate.concurrency` 500 | 40,000 | 0 | count 40,000 | ✅ |
| 88 | BasicTimeSeries | repeat 1 | `--populate` 10000 `--partitions` 100
`--populate.batch` 10 `--threads` 4 | 40,000 | 0 | count 40,000 | ✅ |
| 89 | CountersWide | repeat 1 | `--populate` 5000 `--partitions` 100
`--populate.batch` 10 `--threads` 4 | 20,000 | 0 | sum 20,000 | ✅ |
| 90 | Locking | repeat 1 | `--partitions` 5000 `--threads` 4 | 20,000 | 0 |
count 20,000 | ✅ |
| 91 | KeyValue | repeat 2 | `--populate` all `--partitions` 10000
`--threads` 4 `--populate.concurrency` 500 | 40,000 | 0 | count 40,000 | ✅ |
| 92 | BasicTimeSeries | repeat 2 | `--populate` 10000 `--partitions` 100
`--populate.batch` 10 `--threads` 4 | 40,000 | 0 | count 40,000 | ✅ |
| 93 | CountersWide | repeat 2 | `--populate` 5000 `--partitions` 100
`--populate.batch` 10 `--threads` 4 | 20,000 | 0 | sum 20,000 | ✅ |
| 94 | Locking | repeat 2 | `--partitions` 5000 `--threads` 4 | 20,000 | 0 |
count 20,000 | ✅ |
| 95 | KeyValue | repeat 3 | `--populate` all `--partitions` 10000
`--threads` 4 `--populate.concurrency` 500 | 40,000 | 0 | count 40,000 | ✅ |
| 96 | BasicTimeSeries | repeat 3 | `--populate` 10000 `--partitions` 100
`--populate.batch` 10 `--threads` 4 | 40,000 | 0 | count 40,000 | ✅ |
| 97 | CountersWide | repeat 3 | `--populate` 5000 `--partitions` 100
`--populate.batch` 10 `--threads` 4 | 20,000 | 0 | sum 20,000 | ✅ |
| 98 | Locking | repeat 3 | `--partitions` 5000 `--threads` 4 | 20,000 | 0 |
count 20,000 | ✅ |
| 99 | KeyValue | repeat 4 | `--populate` all `--partitions` 10000
`--threads` 4 `--populate.concurrency` 500 | 40,000 | 0 | count 40,000 | ✅ |
| 100 | BasicTimeSeries | repeat 4 | `--populate` 10000 `--partitions` 100
`--populate.batch` 10 `--threads` 4 | 40,000 | 0 | count 40,000 | ✅ |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
