[
https://issues.apache.org/jira/browse/KAFKA-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15924764#comment-15924764
]
Onur Karaman commented on KAFKA-4896:
-------------------------------------
[~junrao] Naively bumping the number of scheduler threads might cause a
concurrency bug if GroupCoordinator.handleGroupImmigration and
GroupCoordinator.handleGroupEmigration are called concurrently.
handleGroupImmigration schedules a task (doLoadGroupsAndOffsets) that acquires
the partitionLock, updates the loadingPartitions, releases the partitionLock,
loads the metadata into memory, acquires the partitionLock, and updates
ownedPartitions and loadingPartitions, and then releases the partitionLock.
handleGroupEmigration schedules a task (removeGroupsAndOffsets) that acquires
the partitionLock, updates ownedPartitions, clears the metadata from memory,
and then releases the partitionLock.
Adding more threads to the scheduler risks removeGroupsAndOffsets concurrently
clearing metadata while doLoadGroupsAndOffsets is loading metadata by executing
between doLoadGroupsAndOffsets's two windows of holding the partitionLock.
> Offset loading can use more threads
> -----------------------------------
>
> Key: KAFKA-4896
> URL: https://issues.apache.org/jira/browse/KAFKA-4896
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.10.2.0
> Reporter: Jun Rao
> Labels: newbie
>
> Currently, in GroupMetadataManager, we have a single thread for loading the
> offset cache. We could speed it up with more threads.
> /* single-thread scheduler to handle offset/group metadata cache loading and
> unloading */
> private val scheduler = new KafkaScheduler(threads = 1, threadNamePrefix =
> "group-metadata-manager-")
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
