gabriellefu commented on PR #23924: URL: https://github.com/apache/kafka/pull/23924#issuecomment-6638775122
the comparison of this branch change with the balanced assignor on trunk in testbed ### Testbed results Run with the randomized testbed from #23556 (`new TaskAssignorTestbed(new BalancedTaskAssignor())`, default base seed 42) on 9e7183d6be: SMALL has 300 scenarios, LARGE has 30. A cell is `trunk → this PR`, averaged over the scenarios, and a single value means unchanged. **Initial** is the assignment from empty; **rebalance** is the assignment after a random change (1291 SMALL, 131 LARGE), graded against the assignment before the change. LARGE scenarios use only the `region` and `zone` keys. Balance rows are max − min (lower is better), "kept" and "moved" rows are fractions (1.0 = all), and "standby tasks placed without state" is a count per rebalance. Trunk does not read the tags, so its numbers are the same with and without them. **Rack awareness** | metric | SMALL initial | SMALL rebalance | LARGE initial | LARGE rebalance | |---|---|---|---|---| | distinct tag values [region] | 0.786 → 0.876 | 0.790 → 0.875 | 0.762 → 0.882 | 0.762 → 0.881 | | distinct tag values [zone] | 0.742 → 0.838 | 0.769 → 0.854 | 0.785 → 0.887 | 0.782 → 0.890 | | distinct tag values [rack] | 0.756 → 0.854 | 0.782 → 0.873 | n/a | n/a | | tasks lost when one [region] value fails | 0.320 → 0.182 | 0.327 → 0.199 | 0.412 → 0.191 | 0.393 → 0.183 | | tasks lost when one [zone] value fails | 0.421 → 0.253 | 0.362 → 0.221 | 0.390 → 0.194 | 0.392 → 0.186 | | tasks lost when one [rack] value fails | 0.326 → 0.146 | 0.336 → 0.166 | n/a | n/a | The share of stateful tasks that lose their state when one tag value fails drops by 39–55%. **Balance** | metric | SMALL initial | SMALL rebalance | LARGE initial | LARGE rebalance | |---|---|---|---|---| | task balance per member | 1.170 → 1.143 | 1.527 → 1.479 | 1.500 → 1.667 | 1.573 → 1.702 | | load balance per process | 0.729 → 0.691 | 1.085 → 1.055 | 0.968 → 1.197 | 1.086 → 1.286 | | active balance per member | 1.027 → 1.007 | 1.082 → 1.076 | 1.533 | 1.763 → 1.756 | | stateful active balance per member | 0.747 | 0.751 | 1.000 | 1.000 | | active load balance per process | 0.355 | 0.350 | 0.353 | 0.364 | | stateful active load balance per process | 0.299 | 0.320 | 0.376 | 0.411 | | subtopology balance | 0.047 → 0.067 | 0.272 → 0.263 | 0.267 → 0.400 | 1.298 → 1.328 | The active tasks are placed across processes as on trunk. On LARGE, the standbys cost some balance to reach the diversity above: task balance per member +0.13–0.17 and load balance per process +0.20–0.23. SMALL is slightly better than trunk. **Stickiness** | metric | SMALL rebalance | LARGE rebalance | |---|---|---| | stateful active tasks kept on member | 0.608 → 0.607 | 0.462 | | stateful active tasks moved to a process with state | 0.193 → 0.191 | 0.084 | | stateful active tasks moved to a process without state | 0.199 → 0.201 | 0.454 → 0.455 | | active tasks kept on member | 0.492 → 0.488 | 0.369 → 0.366 | | standby tasks placed without state | 2.153 → 2.112 | 341.534 → 342.244 | Unchanged within noise. Rounds to converge and not converged are identical to trunk (see all metrics). **Without rack-aware assignment tags** This PR also deals the stateless active tasks before the standbys when no tags are set (trunk deals them after), so the standbys are placed by the load of all active tasks. The stateless tasks still continue the deal of the stateful ones, so the active placement across processes does not change. | metric | SMALL initial | SMALL rebalance | LARGE initial | LARGE rebalance | |---|---|---|---|---| | task balance per member | 1.170 → 1.080 | 1.527 → 1.405 | 1.500 → 1.300 | 1.573 → 1.313 | | load balance per process | 0.729 → 0.614 | 1.085 → 0.968 | 0.968 → 0.790 | 1.086 → 0.814 | | active balance per member | 1.027 → 1.020 | 1.082 | 1.533 → 1.433 | 1.763 → 1.679 | | subtopology balance | 0.047 → 0.063 | 0.272 → 0.276 | 0.267 → 0.367 | 1.298 → 1.237 | | active tasks kept on member | n/a | 0.492 → 0.489 | n/a | 0.369 → 0.364 | | standby tasks placed without state | n/a | 2.153 → 2.619 | n/a | 341.534 → 377.046 | Task balance per member improves by 8–17% and load balance per process by 11–25%. The cost is more standbys placed without state (+10% on LARGE, +22% on SMALL), since the standbys now follow the stateless deal, which does not look at the previous assignment. The other rows, including the rack rows (here the assignor does not see the tags), move by 0.021 at most. <details> <summary>All metrics, with rack-aware assignment tags</summary> | metric | SMALL initial | SMALL rebalance | LARGE initial | LARGE rebalance | |---|---|---|---|---| | stateful active balance per member | 0.747 | 0.751 | 1.000 | 1.000 | | active balance per member | 1.027 → 1.007 | 1.082 → 1.076 | 1.533 | 1.763 → 1.756 | | task balance per member | 1.170 → 1.143 | 1.527 → 1.479 | 1.500 → 1.667 | 1.573 → 1.702 | | stateful active load balance per process | 0.299 | 0.320 | 0.376 | 0.411 | | active load balance per process | 0.355 | 0.350 | 0.353 | 0.364 | | load balance per process | 0.729 → 0.691 | 1.085 → 1.055 | 0.968 → 1.197 | 1.086 → 1.286 | | subtopology balance | 0.047 → 0.067 | 0.272 → 0.263 | 0.267 → 0.400 | 1.298 → 1.328 | | tasks lost when one [region] value fails | 0.320 → 0.182 | 0.327 → 0.199 | 0.412 → 0.191 | 0.393 → 0.183 | | tasks lost when one [zone] value fails | 0.421 → 0.253 | 0.362 → 0.221 | 0.390 → 0.194 | 0.392 → 0.186 | | tasks lost when one [rack] value fails | 0.326 → 0.146 | 0.336 → 0.166 | n/a | n/a | | distinct tag values [region] | 0.786 → 0.876 | 0.790 → 0.875 | 0.762 → 0.882 | 0.762 → 0.881 | | distinct tag values [zone] | 0.742 → 0.838 | 0.769 → 0.854 | 0.785 → 0.887 | 0.782 → 0.890 | | distinct tag values [rack] | 0.756 → 0.854 | 0.782 → 0.873 | n/a | n/a | | stateful active tasks kept on member | n/a | 0.608 → 0.607 | n/a | 0.462 | | stateful active tasks moved to a process with state | n/a | 0.193 → 0.191 | n/a | 0.084 | | stateful active tasks moved to a process without state | n/a | 0.199 → 0.201 | n/a | 0.454 → 0.455 | | active tasks kept on member | n/a | 0.492 → 0.488 | n/a | 0.369 → 0.366 | | standby tasks placed without state | n/a | 2.153 → 2.112 | n/a | 341.534 → 342.244 | | rounds to converge | 1.000 | 0.990 | 1.000 | 1.000 | | not converged | 0.000 | 0.000 | 0.000 | 0.000 | </details> <details> <summary>All metrics, without rack-aware assignment tags</summary> | metric | SMALL initial | SMALL rebalance | LARGE initial | LARGE rebalance | |---|---|---|---|---| | stateful active balance per member | 0.747 | 0.751 | 1.000 | 1.000 | | active balance per member | 1.027 → 1.020 | 1.082 | 1.533 → 1.433 | 1.763 → 1.679 | | task balance per member | 1.170 → 1.080 | 1.527 → 1.405 | 1.500 → 1.300 | 1.573 → 1.313 | | stateful active load balance per process | 0.299 | 0.320 | 0.376 | 0.411 | | active load balance per process | 0.355 | 0.350 | 0.353 | 0.364 | | load balance per process | 0.729 → 0.614 | 1.085 → 0.968 | 0.968 → 0.790 | 1.086 → 0.814 | | subtopology balance | 0.047 → 0.063 | 0.272 → 0.276 | 0.267 → 0.367 | 1.298 → 1.237 | | tasks lost when one [region] value fails | 0.320 → 0.335 | 0.327 → 0.332 | 0.412 → 0.406 | 0.393 → 0.394 | | tasks lost when one [zone] value fails | 0.421 → 0.409 | 0.362 → 0.351 | 0.390 → 0.396 | 0.392 → 0.384 | | tasks lost when one [rack] value fails | 0.326 → 0.327 | 0.336 → 0.315 | n/a | n/a | | distinct tag values [region] | 0.786 → 0.780 | 0.790 → 0.787 | 0.762 → 0.765 | 0.762 | | distinct tag values [zone] | 0.742 → 0.752 | 0.769 → 0.772 | 0.785 → 0.781 | 0.782 → 0.785 | | distinct tag values [rack] | 0.756 → 0.759 | 0.782 → 0.795 | n/a | n/a | | stateful active tasks kept on member | n/a | 0.608 | n/a | 0.462 → 0.461 | | stateful active tasks moved to a process with state | n/a | 0.193 → 0.192 | n/a | 0.084 | | stateful active tasks moved to a process without state | n/a | 0.199 → 0.200 | n/a | 0.454 | | active tasks kept on member | n/a | 0.492 → 0.489 | n/a | 0.369 → 0.364 | | standby tasks placed without state | n/a | 2.153 → 2.619 | n/a | 341.534 → 377.046 | | rounds to converge | 1.000 | 0.990 | 1.000 | 1.000 | | not converged | 0.000 | 0.000 | 0.000 | 0.000 | </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
