[
https://issues.apache.org/jira/browse/KAFKA-21356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushant Mahajan updated KAFKA-21356:
------------------------------------
Description:
Enabling DLQ on a share group record could strangulate the overall share fetch
throughput by choking the persister. The causes are (in order of priority):
1. `ShareCoordinatorMetadataCacheHelperImpl.isDlqEnabledOnTopic` returns the
value of a dynamic config so it is called repeatedly. The impl of this method
creates a new LogConfig object, on each invocation and parses entire
TopicConfig. On a system with large number of DLQ requests, these calls add up
consuming CPU and choking the persister.
Fix: Use raw config lookup (shrinks cpu samples at high load from 27% to 0.1%)
2. Extra batch reads in `ShareGroupDLQRecordFetcher.collectRecords` - each DLQ
request reads from the source log with maxFetchBytes of 1 MiB, which is about X
batches of Y KB (for example 64 batches of 16KB). The loop in collectRecords
stops only when shouldStop() said the size budget was used up. Currently, we
are looking at all these batches instead of stopping when we have records for
DLQ'ed offsets.
Fix: Discard batches by looking at baseOffset
3. Share group DLQ requires reading data from the ReplicaManagerLogReader,
which is a disk operation. Currently, this happens in the same thread as the
Persister which means that other share coordinator requests remain queued on a
potentially slow disk read operation, again choking the persister.
Fix: Allocate separate executor to the record readers.
was:
Enabling DLQ on a share group record could strangulate the overall share fetch
throughput by choking the persister. The causes are (in order of priority):
1. `ShareCoordinatorMetadataCacheHelperImpl.isDlqEnabledOnTopic` returns the
value of a dynamic config so it is called repeatedly. The impl of this method
creates a new LogConfig object, on each invocation and parses entire
TopicConfig. On a system with large number of DLQ requests, these calls add up
consuming CPU and choking the persister.
Fix: Use raw config lookup (shrinks cpu samples at high load from 27% to 0.1%)
2. Extra batch reads in `ShareGroupDLQRecordFetcher.collectRecords` - each DLQ
request reads from the source log with maxFetchBytes of 1 MiB, which is about X
batches of Y KB (for example 64 batches of 16KB). The loop in collectRecords
stops only when shouldStop() said the size budget was used up. Currently, we
are looking at all these batches instead of stopping when we have records for
DLQ'ed offsets.
3. Share group DLQ requires reading data from the ReplicaManagerLogReader,
which is a disk operation. Currently, this happens in the same thread as the
Persister which means that other share coordinator requests remain queued on a
potentially slow disk read operation, again choking the persister.
Fix: Allocate separate executor to the record readers.
> Performance issue in share fetch with share group DLQ enabled.
> --------------------------------------------------------------
>
> Key: KAFKA-21356
> URL: https://issues.apache.org/jira/browse/KAFKA-21356
> Project: Kafka
> Issue Type: Bug
> Reporter: Sushant Mahajan
> Assignee: Sushant Mahajan
> Priority: Major
>
> Enabling DLQ on a share group record could strangulate the overall share
> fetch throughput by choking the persister. The causes are (in order of
> priority):
> 1. `ShareCoordinatorMetadataCacheHelperImpl.isDlqEnabledOnTopic` returns the
> value of a dynamic config so it is called repeatedly. The impl of this method
> creates a new LogConfig object, on each invocation and parses entire
> TopicConfig. On a system with large number of DLQ requests, these calls add
> up consuming CPU and choking the persister.
> Fix: Use raw config lookup (shrinks cpu samples at high load from 27% to 0.1%)
> 2. Extra batch reads in `ShareGroupDLQRecordFetcher.collectRecords` - each
> DLQ request reads from the source log with maxFetchBytes of 1 MiB, which is
> about X batches of Y KB (for example 64 batches of 16KB). The loop in
> collectRecords stops only when shouldStop() said the size budget was used up.
> Currently, we are looking at all these batches instead of stopping when we
> have records for DLQ'ed offsets.
> Fix: Discard batches by looking at baseOffset
> 3. Share group DLQ requires reading data from the ReplicaManagerLogReader,
> which is a disk operation. Currently, this happens in the same thread as the
> Persister which means that other share coordinator requests remain queued on
> a potentially slow disk read operation, again choking the persister.
> Fix: Allocate separate executor to the record readers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
