suzhiking opened a new pull request, #23969: URL: https://github.com/apache/kafka/pull/23969
`AssignmentRefinerImpl.refine()` runs on every streams group heartbeat. This PR makes a call cheaper without changing the intermediate assignment it derives. Each commit is one optimization and can be reviewed on its own: 1. Return the target assignment right after the case analysis when it stages no migration. 2. Walk every task fewer times: count process load in the index walk, find borrowed standby conflicts through the target standby holders, and build that lookup only when it is read. 3. Sort only the decisions, not every stateful target active task. 4. Skip counting active tasks when the refiner returns the target assignment itself. 5. Resolve the stateful subtopologies once per call instead of per (member, subtopology) entry. 6. Find the tasks not on their target owner first, and return before indexing when there are none. This assumes the target assignment names one member per active task, as the rest of the refiner already does. 7. Compare task IDs directly instead of building a comparator chain per comparison. 8. Index only the tasks not on their target owner; the process load still counts every task. 9. Build the target standby holders only for those tasks. 10. Patch members copy-on-write instead of copying every patched member in full. 11. Keep borrowed and parked migrations in hash sets instead of tree sets. 12. Count active tasks only for members whose tasks differ from the target assignment. 13. Build the state-on-disk lookup only when the case analysis asks for it. Besides the unit tests, each step was checked against the previous refiner on 20,000 random groups, with identical results. `AssignmentRefinerBenchmark.refinerImpl`, 1,000 members, 5,000 stateful tasks, one standby, MIXED lag, budget 2, through step 10, µs per call: | Migrating | Before | After | |---|---|---| | 0% | 4,573 | 163 | | 1% | 4,760 | 1,240 | | 50% | 6,855 | 3,365 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
