gabriellefu commented on PR #23924:
URL: https://github.com/apache/kafka/pull/23924#issuecomment-6639315374
### JMH results
Run with `StreamsAssignorBenchmark` (`assignorType=BALANCED`) on 9e7183d6be,
and with the same benchmark on trunk's `BalancedTaskAssignor`, for the four tag
layouts only. Trunk does not read the tags, so its numbers are the tag-blind
cost of the same group. 1000 members, 100 subtopologies with 100 partitions
each (10,000 tasks), no reported offsets, 1 fork, 3 × 2s warmup, 5 × 2s
measurement. A cell is the average time of one assignment in ms, `trunk → this
PR`, with the error where it is above 10% of the score. **Full** is the
assignment from empty; **incremental** is the assignment after one member
joins. Each process takes the value of every tag key round-robin, and `host`
has a value of its own per process.
**1000 processes (1 member per process)**
| rack.aware.assignment.tags | full, 1 standby | full, 2 standbys |
incremental, 1 standby | incremental, 2 standbys |
|---|---|---|---|---|
| cluster, zone | 9.65 → 13.89 | 21.14 → 24.39 | 10.22 → 14.89 | 21.70 →
21.12 |
| cluster, zone, rack | 9.61 → 11.40 | 21.74 → 23.12 | 10.11 → 12.43 | 24.35
→ 21.75 ± 2.66 |
| cluster, zone, host | 9.32 → 9.59 | 21.91 → 25.02 | 10.22 → 10.60 | 24.32
→ 25.05 |
| region, cluster, zone, rack, host | 9.25 → 10.13 | 20.28 → 21.67 | 10.30 →
10.89 | 23.22 → 28.59 |
**20 processes (50 members per process)**
| rack.aware.assignment.tags | full, 1 standby | full, 2 standbys |
incremental, 1 standby | incremental, 2 standbys |
|---|---|---|---|---|
| cluster, zone | 3.78 → 3.32 | 5.64 → 5.13 | 4.04 ± 0.41 → 3.97 | 8.48 →
6.22 |
| cluster, zone, rack | 3.71 → 3.55 | 8.69 → 5.35 | 7.05 → 4.30 | 8.65 →
6.38 |
| cluster, zone, host | 3.69 → 3.46 | 8.72 → 5.20 | 4.04 → 4.16 | 9.85 →
6.42 |
| region, cluster, zone, rack, host | 3.72 → 3.80 | 8.70 → 5.81 | 6.92 →
4.51 | 6.07 → 7.21 |
Since trunk does the same work for every tag layout, its four rows in a
column show the run-to-run noise: up to 12% apart with 1000 processes, but up
to 1.7× apart with 20 processes (for example 4.04 and 7.05 ms for incremental
with 1 standby).
With 1000 single-member processes and 1 standby, `cluster, zone` costs the
most (+44–46%, 13.9–14.9 ms), `cluster, zone, rack` +19–23%, and the layouts
with a host key +3–9%. With 2 standbys the change is −11% to +23%, and the
slowest assignment takes 28.6 ms (incremental, five keys; 23.2 ms on trunk).
With 20 processes this PR takes 3.3–7.2 ms against 3.7–9.8 ms on trunk; trunk's
spread there is too large for one fork to show a difference.
<details>
<summary>All results</summary>
```
./jmh-benchmarks/jmh.sh 'StreamsAssignor(Trunk)?Benchmark\.doAssignment' \
-f 1 -wi 3 -w 2s -i 5 -r 2s \
-p assignorType=BALANCED -p memberCount=1000 -p partitionCount=100 -p
subtopologyCount=100 \
-p reportedOffsets=NONE -p membersPerProcess=1,50 -p
assignmentType=FULL,INCREMENTAL -p standbyReplicas=1,2 \
-p
rackAwareTags=CLUSTER_AND_ZONE,CLUSTER_ZONE_AND_RACK,CLUSTER_ZONE_AND_HOST,REGION_CLUSTER_ZONE_RACK_AND_HOST
```
`StreamsAssignorTrunkBenchmark` is a local copy of
`StreamsAssignorBenchmark` that runs trunk's `BalancedTaskAssignor`. Score ±
error in ms.
```
members per process type standbys rackAwareTags
side ms/op
1 FULL 1 CLUSTER_AND_ZONE
trunk 9.646 ± 0.427
1 FULL 1 CLUSTER_AND_ZONE
this PR 13.888 ± 0.441
1 FULL 1 CLUSTER_ZONE_AND_RACK
trunk 9.609 ± 0.387
1 FULL 1 CLUSTER_ZONE_AND_RACK
this PR 11.400 ± 0.607
1 FULL 1 CLUSTER_ZONE_AND_HOST
trunk 9.315 ± 0.299
1 FULL 1 CLUSTER_ZONE_AND_HOST
this PR 9.595 ± 0.471
1 FULL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 9.255 ± 0.421
1 FULL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 10.128 ± 0.533
1 FULL 2 CLUSTER_AND_ZONE
trunk 21.139 ± 0.757
1 FULL 2 CLUSTER_AND_ZONE
this PR 24.386 ± 1.817
1 FULL 2 CLUSTER_ZONE_AND_RACK
trunk 21.737 ± 1.436
1 FULL 2 CLUSTER_ZONE_AND_RACK
this PR 23.117 ± 1.072
1 FULL 2 CLUSTER_ZONE_AND_HOST
trunk 21.911 ± 0.699
1 FULL 2 CLUSTER_ZONE_AND_HOST
this PR 25.016 ± 1.224
1 FULL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 20.283 ± 0.857
1 FULL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 21.674 ± 1.834
1 INCREMENTAL 1 CLUSTER_AND_ZONE
trunk 10.222 ± 0.418
1 INCREMENTAL 1 CLUSTER_AND_ZONE
this PR 14.892 ± 0.205
1 INCREMENTAL 1 CLUSTER_ZONE_AND_RACK
trunk 10.108 ± 0.530
1 INCREMENTAL 1 CLUSTER_ZONE_AND_RACK
this PR 12.434 ± 0.042
1 INCREMENTAL 1 CLUSTER_ZONE_AND_HOST
trunk 10.224 ± 0.309
1 INCREMENTAL 1 CLUSTER_ZONE_AND_HOST
this PR 10.596 ± 0.127
1 INCREMENTAL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 10.303 ± 0.246
1 INCREMENTAL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 10.888 ± 0.111
1 INCREMENTAL 2 CLUSTER_AND_ZONE
trunk 21.704 ± 1.033
1 INCREMENTAL 2 CLUSTER_AND_ZONE
this PR 21.117 ± 1.173
1 INCREMENTAL 2 CLUSTER_ZONE_AND_RACK
trunk 24.348 ± 1.255
1 INCREMENTAL 2 CLUSTER_ZONE_AND_RACK
this PR 21.745 ± 2.661
1 INCREMENTAL 2 CLUSTER_ZONE_AND_HOST
trunk 24.319 ± 2.018
1 INCREMENTAL 2 CLUSTER_ZONE_AND_HOST
this PR 25.053 ± 1.879
1 INCREMENTAL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 23.217 ± 1.284
1 INCREMENTAL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 28.588 ± 0.916
50 FULL 1 CLUSTER_AND_ZONE
trunk 3.781 ± 0.085
50 FULL 1 CLUSTER_AND_ZONE
this PR 3.321 ± 0.200
50 FULL 1 CLUSTER_ZONE_AND_RACK
trunk 3.706 ± 0.054
50 FULL 1 CLUSTER_ZONE_AND_RACK
this PR 3.548 ± 0.143
50 FULL 1 CLUSTER_ZONE_AND_HOST
trunk 3.688 ± 0.022
50 FULL 1 CLUSTER_ZONE_AND_HOST
this PR 3.462 ± 0.124
50 FULL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 3.716 ± 0.051
50 FULL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 3.801 ± 0.172
50 FULL 2 CLUSTER_AND_ZONE
trunk 5.636 ± 0.205
50 FULL 2 CLUSTER_AND_ZONE
this PR 5.126 ± 0.160
50 FULL 2 CLUSTER_ZONE_AND_RACK
trunk 8.692 ± 0.196
50 FULL 2 CLUSTER_ZONE_AND_RACK
this PR 5.349 ± 0.202
50 FULL 2 CLUSTER_ZONE_AND_HOST
trunk 8.722 ± 0.107
50 FULL 2 CLUSTER_ZONE_AND_HOST
this PR 5.203 ± 0.215
50 FULL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 8.700 ± 0.127
50 FULL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 5.814 ± 0.333
50 INCREMENTAL 1 CLUSTER_AND_ZONE
trunk 4.042 ± 0.409
50 INCREMENTAL 1 CLUSTER_AND_ZONE
this PR 3.966 ± 0.077
50 INCREMENTAL 1 CLUSTER_ZONE_AND_RACK
trunk 7.048 ± 0.303
50 INCREMENTAL 1 CLUSTER_ZONE_AND_RACK
this PR 4.298 ± 0.300
50 INCREMENTAL 1 CLUSTER_ZONE_AND_HOST
trunk 4.042 ± 0.132
50 INCREMENTAL 1 CLUSTER_ZONE_AND_HOST
this PR 4.156 ± 0.282
50 INCREMENTAL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 6.925 ± 0.253
50 INCREMENTAL 1 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 4.506 ± 0.163
50 INCREMENTAL 2 CLUSTER_AND_ZONE
trunk 8.482 ± 0.247
50 INCREMENTAL 2 CLUSTER_AND_ZONE
this PR 6.221 ± 0.295
50 INCREMENTAL 2 CLUSTER_ZONE_AND_RACK
trunk 8.647 ± 0.518
50 INCREMENTAL 2 CLUSTER_ZONE_AND_RACK
this PR 6.382 ± 0.175
50 INCREMENTAL 2 CLUSTER_ZONE_AND_HOST
trunk 9.848 ± 0.231
50 INCREMENTAL 2 CLUSTER_ZONE_AND_HOST
this PR 6.417 ± 0.368
50 INCREMENTAL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
trunk 6.070 ± 0.241
50 INCREMENTAL 2 REGION_CLUSTER_ZONE_RACK_AND_HOST
this PR 7.213 ± 0.286
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
