[
https://issues.apache.org/jira/browse/KAFKA-20917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Wang updated KAFKA-20917:
----------------------------
Description:
*Latency regression after upgrading from 2.8 to 3.9*
After upgrading producer client from 2.8 to 3.9, we noticed produce latency
increase, e.g. for ack-all producer, P50 latency increase from 8ms to 13ms,
~60% increase.
!Screenshot 2026-08-19 at 2.49.09 PM.png|width=923,height=257!
*What changed?*
The profiling shows RecordAccumulator.ready call is hot. 3.9 introduced
adaptive partitioning and added more checks and calculations per topic
partition in RecordAccumulator.ready -> partitionReady method.
*2.8 client profiling*
!Screenshot 2026-08-19 at 2.29.20 PM.png|width=916,height=240!
low cluster.leaderFor call.
!Screenshot 2026-08-19 at 2.34.29 PM.png|width=912,height=205!
no metadataSnapshot.leaderEpochFor call (introduced in 3.9)
*3.9 client profiling* {*}({*}{*}With adaptive partitioning disabled){*}{*}{{*}}
heavy metadataSnapshot.leaderEpochFor and cluster.leaderFor calls
!Screenshot 2026-08-19 at 2.31.36 PM.png|width=906,height=273!
!Screenshot 2026-08-19 at 2.38.46 PM.png|width=906,height=228!
!Screenshot 2026-08-19 at 2.40.28 PM.png|width=897,height=270!
*After Fix*
Check the fix in the linked PRs.
{*}3.9 client with fix profiling ({*}{*}With adaptive partitioning disabled){*}
*!Screenshot 2026-08-19 at 2.43.46 PM.png|width=885,height=294!*
reduce the cluster.leaderFor call to similar as 2.9 client, no noticeable
metadataSnapshot.leaderEpochFor call.
*!Screenshot 2026-08-19 at 2.45.06 PM.png|width=884,height=240!*
latency reduced to 8ms, similar to 2.8 client latency.
!Screenshot 2026-08-19 at 3.00.26 PM.png|width=884,height=240!
was:
*Latency regression after upgrading from 2.8 to 3.9*
After upgrading producer client from 2.8 to 3.9, we noticed produce latency
increase, e.g. for ack-all producer, P50 latency increase from 8ms to 13ms,
~60% increase.
!Screenshot 2026-08-19 at 2.49.09 PM.png|width=923,height=257!
*What changed?*
The profiling shows RecordAccumulator.ready call is hot. 3.9 introduced
adaptive partitioning and added more checks and calculations per topic
partition in RecordAccumulator.ready -> partitionReady method.
*2.8 client profiling*
!Screenshot 2026-08-19 at 2.29.20 PM.png|width=916,height=240!
low cluster.leaderFor call.
!Screenshot 2026-08-19 at 2.34.29 PM.png|width=912,height=205!
no metadataSnapshot.leaderEpochFor call (introduced in 3.9)
*3.9 client profiling* {*}({*}{*}With adaptive partitioning disabled){*}{*}{{*}}
heavy metadataSnapshot.leaderEpochFor and cluster.leaderFor calls
!Screenshot 2026-08-19 at 2.31.36 PM.png|width=906,height=273!
!Screenshot 2026-08-19 at 2.38.46 PM.png|width=906,height=228!
!Screenshot 2026-08-19 at 2.40.28 PM.png|width=897,height=270!
*After Fix*
{*}3.9 client with fix profiling ({*}{*}With adaptive partitioning disabled){*}
*!Screenshot 2026-08-19 at 2.43.46 PM.png|width=885,height=294!*
reduce the cluster.leaderFor call to similar as 2.9 client, no noticeable
metadataSnapshot.leaderEpochFor call.
*!Screenshot 2026-08-19 at 2.45.06 PM.png|width=884,height=240!*
latency reduced to 8ms, similar to 2.8 client latency.
!Screenshot 2026-08-19 at 3.00.26 PM.png|width=884,height=240!
> [Java-client] RecordAccumulator.ready performance regression
> ------------------------------------------------------------
>
> Key: KAFKA-20917
> URL: https://issues.apache.org/jira/browse/KAFKA-20917
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Reporter: Xi Wang
> Priority: Major
> Attachments: Screenshot 2026-07-02 at 1.59.07 PM-1.png, Screenshot
> 2026-07-02 at 1.59.07 PM.png, Screenshot 2026-08-19 at 2.29.20 PM.png,
> Screenshot 2026-08-19 at 2.31.36 PM.png, Screenshot 2026-08-19 at 2.34.29
> PM.png, Screenshot 2026-08-19 at 2.38.46 PM.png, Screenshot 2026-08-19 at
> 2.40.28 PM.png, Screenshot 2026-08-19 at 2.43.46 PM.png, Screenshot
> 2026-08-19 at 2.45.06 PM.png, Screenshot 2026-08-19 at 2.49.09 PM.png,
> Screenshot 2026-08-19 at 3.00.26 PM.png
>
>
> *Latency regression after upgrading from 2.8 to 3.9*
> After upgrading producer client from 2.8 to 3.9, we noticed produce latency
> increase, e.g. for ack-all producer, P50 latency increase from 8ms to 13ms,
> ~60% increase.
> !Screenshot 2026-08-19 at 2.49.09 PM.png|width=923,height=257!
> *What changed?*
> The profiling shows RecordAccumulator.ready call is hot. 3.9 introduced
> adaptive partitioning and added more checks and calculations per topic
> partition in RecordAccumulator.ready -> partitionReady method.
> *2.8 client profiling*
> !Screenshot 2026-08-19 at 2.29.20 PM.png|width=916,height=240!
> low cluster.leaderFor call.
> !Screenshot 2026-08-19 at 2.34.29 PM.png|width=912,height=205!
> no metadataSnapshot.leaderEpochFor call (introduced in 3.9)
> *3.9 client profiling* {*}({*}{*}With adaptive partitioning
> disabled){*}{*}{{*}}
> heavy metadataSnapshot.leaderEpochFor and cluster.leaderFor calls
> !Screenshot 2026-08-19 at 2.31.36 PM.png|width=906,height=273!
> !Screenshot 2026-08-19 at 2.38.46 PM.png|width=906,height=228!
> !Screenshot 2026-08-19 at 2.40.28 PM.png|width=897,height=270!
> *After Fix*
> Check the fix in the linked PRs.
> {*}3.9 client with fix profiling ({*}{*}With adaptive partitioning
> disabled){*}
> *!Screenshot 2026-08-19 at 2.43.46 PM.png|width=885,height=294!*
> reduce the cluster.leaderFor call to similar as 2.9 client, no noticeable
> metadataSnapshot.leaderEpochFor call.
> *!Screenshot 2026-08-19 at 2.45.06 PM.png|width=884,height=240!*
> latency reduced to 8ms, similar to 2.8 client latency.
> !Screenshot 2026-08-19 at 3.00.26 PM.png|width=884,height=240!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
