artemlivshits opened a new pull request, #12049: URL: https://github.com/apache/kafka/pull/12049
The design is described in detail in KIP-794 https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner. Implementation notes: The default partitioning logic is moved to the BuiltInPartitioner class (there is one object per topic). The object keeps track of how many bytes are produced per-partition and once the amount exceeds batch.size, switches to the next partition (note that partition switch decision is decoupled from batching). The object also keeps track of probability weights that are based on the queue sizes (the larger the queue size is the less chance for the next partition to be chosen). The queue sizes are calculated in the RecordAccumulator in the `ready` method, the method already enumerates all partitions so we just add some extra logic into the existing O(N) method. The partition switch decision may take O(logN), where N is the number partitions per topic, but it happens only once per batch.size (and the logic is avoided when all queues are of equal size). Produce bytes accounting logic is lock-free. When partitioner.availability.timeout.ms is non-0, RecordAccumulator keeps stats on "node latency" which is defined as the difference between the last time the node had a batch waiting to be send and the last time the node was ready to take a new batch. If this difference exceeds partitioner.availability.timeout.ms we don't switch to that partition until the node is ready. The corresponding unit tests are added / modified. The perf test results are in the KIP-794. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
