[
https://issues.apache.org/jira/browse/KAFKA-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas Telford updated KAFKA-15178:
-------------------------------------
Description:
Doing some profiling of my Kafka Streams application, I noticed that the
{{pollPhase}} suffers from a minor performance issue.
See the pink tree on the left of the flame graph below.
!pollPhase.png|width=1028,height=308!
{{ConsumerCoordinator.poll}} calls {{{}rejoinNeededOrPending{}}}, which checks
the current {{metadataSnapshot}} against the {{{}assignmentSnapshot{}}}. This
comparison is a deep-equality check, and if there's a large number of
topic-partitions being consumed by the application, then this comparison can
perform poorly.
I suspect this can be trivially addressed with a {{boolean}} flag that
indicates when the {{metadataSnapshot}} has been updated (or is "dirty"), and
actually needs to be checked, since most of the time it should be identical to
{{{}assignmentSnapshot{}}}.
I plan to raise a PR with this optimization to address this issue.
was:
Doing some profiling of my Kafka Streams application, I noticed that the
{{pollPhase}} suffers from a minor performance issue.
See flame graph below. !pollPhase.png|width=1028,height=308!
{{ConsumerCoordinator.poll}} calls {{{}rejoinNeededOrPending{}}}, which checks
the current {{metadataSnapshot}} against the {{{}assignmentSnapshot{}}}. This
comparison is a deep-equality check, and if there's a large number of
topic-partitions being consumed by the application, then this comparison can
perform poorly.
I suspect this can be trivially addressed with a {{boolean}} flag that
indicates when the {{metadataSnapshot}} has been updated (or is "dirty"), and
actually needs to be checked, since most of the time it should be identical to
{{{}assignmentSnapshot{}}}.
I plan to raise a PR with this optimization to address this issue.
> Poor performance of ConsumerCoordinator with many TopicPartitions
> -----------------------------------------------------------------
>
> Key: KAFKA-15178
> URL: https://issues.apache.org/jira/browse/KAFKA-15178
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 3.5.0
> Reporter: Nicholas Telford
> Assignee: Nicholas Telford
> Priority: Minor
> Labels: easyfix
> Attachments: pollPhase.png
>
>
> Doing some profiling of my Kafka Streams application, I noticed that the
> {{pollPhase}} suffers from a minor performance issue.
> See the pink tree on the left of the flame graph below.
> !pollPhase.png|width=1028,height=308!
> {{ConsumerCoordinator.poll}} calls {{{}rejoinNeededOrPending{}}}, which
> checks the current {{metadataSnapshot}} against the
> {{{}assignmentSnapshot{}}}. This comparison is a deep-equality check, and if
> there's a large number of topic-partitions being consumed by the application,
> then this comparison can perform poorly.
> I suspect this can be trivially addressed with a {{boolean}} flag that
> indicates when the {{metadataSnapshot}} has been updated (or is "dirty"), and
> actually needs to be checked, since most of the time it should be identical
> to {{{}assignmentSnapshot{}}}.
> I plan to raise a PR with this optimization to address this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
