Jackie-Jiang opened a new issue, #16314:
URL: https://github.com/apache/pinot/issues/16314
`KafkaConsumer` cannot be used in a multi-threaded manner
```
2025/07/09 22:30:06.371 WARN [RealtimeSegmentDataManager_*] [Start a Pinot
[SERVER]] Cannot fetch stream offset with criteria
OffsetCriteria{_offsetType=LARGEST, _offsetString='largest'} for clientId * and
partitionGroupId 16 with maxWaitTime 5000
java.util.ConcurrentModificationException: KafkaConsumer is not safe for
multi-threaded access
at
org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:2445)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2429)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.kafka.clients.consumer.KafkaConsumer.endOffsets(KafkaConsumer.java:2213)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchStreamPartitionOffset(KafkaStreamMetadataProvider.java:103)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchStreamOffset(RealtimeSegmentDataManager.java:1778)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchLatestStreamOffset(RealtimeSegmentDataManager.java:1753)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.fetchLatestStreamOffset(RealtimeSegmentDataManager.java:1758)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.server.starter.helix.FreshnessBasedConsumptionStatusChecker.isSegmentCaughtUp(FreshnessBasedConsumptionStatusChecker.java:76)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.server.starter.helix.IngestionBasedConsumptionStatusChecker.getNumConsumingSegmentsNotReachedIngestionCriteria(IngestionBasedConsumptionStatusChecker.java:96)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.common.utils.ServiceStatus$RealtimeConsumptionCatchupServiceStatusCallback.getServiceStatus(ServiceStatus.java:277)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
at
org.apache.pinot.common.utils.ServiceStatus$MultipleCallbackServiceStatusCallback.getServiceStatus(ServiceStatus.java:179)
~[startree-pinot-all-1.4.0-ST.21.10-jar-with-dependencies.jar:1.4.0-ST.21.10-9ea7462b2f105a4a3547c74480747205e9a7b9f9]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
