capistrant opened a new pull request, #20793:
URL: https://github.com/apache/druid/pull/20793
### Description
Add a multi segment benchmark for MSQ that can run both on the task engine
and with Dart. This adds a new baseline benchmark that measures both query
runtime as well as counters for segments, rows read, written, etc. I'm going to
be introducing a proposed change for time ordered limit efficiencies for scan
queries in the near future and these benchmarks serve as some before/after
evidence for value. Pushing as an isolated PR for ease of review for both this
and the forthcoming improvements.
#### Data
`DartScanBenchmark` on master: 30 one-day time chunks × 4 segments (120
segments × 100,000 rows = 12M rows), `clusteringCardinality=100`,
`maxNumTasks=2`.
```
Benchmark
(clusteringCardinality) (layout) (limit) (maxNumTasks)
(numPartitionsPerTimeChunk) (numTimeChunks) (rowsPerSegment) Mode Cnt
Score Error Units
DartScanBenchmark.clusterFilteredTimeDescLimit
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 13.141 ±
18.634 ms/op
DartScanBenchmark.clusterFilteredTimeDescLimit:gc.alloc.rate.norm
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 18252340.508 ±
741523.263 B/op
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsInAcquiredSegments
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsWritten
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 1200.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:segmentsAcquired
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 115.783 ±
61.081 ms/op
DartScanBenchmark.clusterFilteredTimeDescLimit:gc.alloc.rate.norm
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 91301251.110 ±
2948451.610 B/op
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsInAcquiredSegments
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsWritten
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 121035.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:segmentsAcquired
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 18.272 ±
30.680 ms/op
DartScanBenchmark.clusterFilteredTimeDescLimit:gc.alloc.rate.norm
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 36002433.371 ±
2982323.533 B/op
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsInAcquiredSegments
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsWritten
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 1200.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:segmentsAcquired
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 58.817 ±
42.928 ms/op
DartScanBenchmark.clusterFilteredTimeDescLimit:gc.alloc.rate.norm
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 127807681.837 ±
2617736.955 B/op
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsInAcquiredSegments
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:rowsWritten
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 121035.000
#
DartScanBenchmark.clusterFilteredTimeDescLimit:segmentsAcquired
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescFilteredNoLimit
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 222.063 ±
95.675 ms/op
DartScanBenchmark.timeDescFilteredNoLimit:gc.alloc.rate.norm
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 413538330.184 ±
2586531.842 B/op
DartScanBenchmark.timeDescFilteredNoLimit:rowsInAcquiredSegments
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescFilteredNoLimit:rowsWritten
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 267502.000
#
DartScanBenchmark.timeDescFilteredNoLimit:segmentsAcquired
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescFilteredNoLimit
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 232.373 ±
125.544 ms/op
DartScanBenchmark.timeDescFilteredNoLimit:gc.alloc.rate.norm
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 413630543.756 ±
2961945.257 B/op
DartScanBenchmark.timeDescFilteredNoLimit:rowsInAcquiredSegments
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescFilteredNoLimit:rowsWritten
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 267502.000
#
DartScanBenchmark.timeDescFilteredNoLimit:segmentsAcquired
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescFilteredNoLimit
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 831.740 ±
487.763 ms/op
DartScanBenchmark.timeDescFilteredNoLimit:gc.alloc.rate.norm
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 2292402366.222 ±
3167092.821 B/op
DartScanBenchmark.timeDescFilteredNoLimit:rowsInAcquiredSegments
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescFilteredNoLimit:rowsWritten
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 267502.000
#
DartScanBenchmark.timeDescFilteredNoLimit:segmentsAcquired
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescFilteredNoLimit
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 770.971 ±
248.374 ms/op
DartScanBenchmark.timeDescFilteredNoLimit:gc.alloc.rate.norm
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 2296423086.857 ±
3460054.644 B/op
DartScanBenchmark.timeDescFilteredNoLimit:rowsInAcquiredSegments
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescFilteredNoLimit:rowsWritten
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 267502.000
#
DartScanBenchmark.timeDescFilteredNoLimit:segmentsAcquired
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescLimit
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 12.375 ±
9.486 ms/op
DartScanBenchmark.timeDescLimit:gc.alloc.rate.norm
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 17496281.961 ±
403318.816 B/op
DartScanBenchmark.timeDescLimit:rowsInAcquiredSegments
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescLimit:rowsWritten
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 1200.000
#
DartScanBenchmark.timeDescLimit:segmentsAcquired
100 V9_TIME_ORDERED 10 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescLimit
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 296.455 ±
44.040 ms/op
DartScanBenchmark.timeDescLimit:gc.alloc.rate.norm
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 601362622.431 ±
1937404.246 B/op
DartScanBenchmark.timeDescLimit:rowsInAcquiredSegments
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescLimit:rowsWritten
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 1200000.000
#
DartScanBenchmark.timeDescLimit:segmentsAcquired
100 V9_TIME_ORDERED 10000 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescLimit
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 495.014 ±
143.856 ms/op
DartScanBenchmark.timeDescLimit:gc.alloc.rate.norm
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 1594349718.885 ±
1993320.011 B/op
DartScanBenchmark.timeDescLimit:rowsInAcquiredSegments
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescLimit:rowsWritten
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 1200.000
#
DartScanBenchmark.timeDescLimit:segmentsAcquired
100 V10_CLUSTERED 10 2
4 30 100000 avgt 3 120.000
#
DartScanBenchmark.timeDescLimit
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 1100.912 ±
197.168 ms/op
DartScanBenchmark.timeDescLimit:gc.alloc.rate.norm
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 2433361348.267 ±
7074383.576 B/op
DartScanBenchmark.timeDescLimit:rowsInAcquiredSegments
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 12000000.000
#
DartScanBenchmark.timeDescLimit:rowsWritten
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 1200000.000
#
DartScanBenchmark.timeDescLimit:segmentsAcquired
100 V10_CLUSTERED 10000 2
4 30 100000 avgt 3 120.000
#
```
below is simplified output in markdown table. With my forthcoming work on
time ordered scans with limit, the counters will be clear and obvious areas
where the work is reduced for some of these benchmark queries who take the
optimized path.
| Query | Layout | Limit | ms/op (± 99.9% CI) | Alloc / query | Segments |
Rows in segments | Rows written |
|---|---|---:|---:|---:|---:|---:|---:|
| `clusterFilteredTimeDescLimit` | V9_TIME_ORDERED | 10 | 13.1 ± 18.6 | 18.3
MB | 120 | 12,000,000 | 1,200 |
| | V9_TIME_ORDERED | 10000 | 115.8 ± 61.1 | 91.3 MB | 120 | 12,000,000 |
121,035 |
| | V10_CLUSTERED | 10 | 18.3 ± 30.7 | 36.0 MB | 120 | 12,000,000 | 1,200 |
| | V10_CLUSTERED | 10000 | 58.8 ± 42.9 | 127.8 MB | 120 | 12,000,000 |
121,035 |
| `timeDescFilteredNoLimit` | V9_TIME_ORDERED | 10 | 222.1 ± 95.7 | 413.5 MB
| 120 | 12,000,000 | 267,502 |
| | V9_TIME_ORDERED | 10000 | 232.4 ± 125.5 | 413.6 MB | 120 | 12,000,000 |
267,502 |
| | V10_CLUSTERED | 10 | 831.7 ± 487.8 | 2,292.4 MB | 120 | 12,000,000 |
267,502 |
| | V10_CLUSTERED | 10000 | 771.0 ± 248.4 | 2,296.4 MB | 120 | 12,000,000 |
267,502 |
| `timeDescLimit` | V9_TIME_ORDERED | 10 | 12.4 ± 9.5 | 17.5 MB | 120 |
12,000,000 | 1,200 |
| | V9_TIME_ORDERED | 10000 | 296.5 ± 44.0 | 601.4 MB | 120 | 12,000,000 |
1,200,000 |
| | V10_CLUSTERED | 10 | 495.0 ± 143.9 | 1,594.3 MB | 120 | 12,000,000 |
1,200 |
| | V10_CLUSTERED | 10000 | 1,100.9 ± 197.2 | 2,433.4 MB | 120 | 12,000,000
| 1,200,000 |
- **ms/op**: average time of one query, from submitting the SQL to
collecting the query report, with the controller and workers in one JVM.
- **Alloc / query**: bytes allocated per query (`gc.alloc.rate.norm`), in MB
(10^6 bytes).
- **Segments / Rows in segments**: segments the scan stage acquired, and the
total rows in them.
- **Rows written**: rows the scan stage wrote to its output, summed over
workers.
- `timeDescFilteredNoLimit` has no limit, so its two `limit` rows run the
same query.
#### Release note
N/A
<hr>
##### Key changed/added classes in this PR
* all of em
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not
all of these items apply to every PR. Remove the items which are not done or
not relevant to the PR. None of the items from the checklist below are strictly
necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
