Eywek opened a new issue, #19963: URL: https://github.com/apache/pulsar/issues/19963
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version 2.7.2 see [Apache Pinot Apache Pulsar stream ingestion plugin pom.xml](https://github.com/apache/pinot/blob/0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e/pinot-plugins/pinot-stream-ingestion/pinot-pulsar/pom.xml#L40) on Debian bullseye ### Minimal reproduce step Hard to reproduce but we're using Apache Pinot [Apache Pulsar stream ingestion plugin](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/apache-pulsar) to retrieve messages from Pulsar (see implementation code [here](https://github.com/apache/pinot/blob/0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e/pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarPartitionLevelConsumer.java#L73-L96)). We're using the commit https://github.com/apache/pinot/tree/0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e for Apache Pinot. ### What did you expect to see? No error in logs ### What did you see instead? ``` Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: 2023/03/29 08:50:57.822 ERROR [LLRealtimeSegmentDataManager_worker_datasource_636bd9305a9600010051fb4d_641de187ab174a000713364a_35__0__8__20230329T0846Z] [worker_datasource_636bd9305a9600010051fb4d_641de187ab174a000713364a_35__0__8__20230329T0846Z] Exception while in work Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: java.util.ConcurrentModificationException: null Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at java.util.ArrayList$Itr.checkForComodification(Unknown Source) ~[?:?] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at java.util.ArrayList$Itr.next(Unknown Source) ~[?:?] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at org.apache.pinot.shaded.com.google.common.collect.Iterators$6.computeNext(Iterators.java:616) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at org.apache.pinot.shaded.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at org.apache.pinot.shaded.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at java.lang.Iterable.forEach(Unknown Source) ~[?:?] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at org.apache.pinot.plugin.stream.pulsar.PulsarMessageBatch.<init>(PulsarMessageBatch.java:50) ~[pinot-pulsar-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-68b4096745272706af3e09f08ecbbd89eac7e3a3] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at org.apache.pinot.plugin.stream.pulsar.PulsarPartitionLevelConsumer.fetchMessages(PulsarPartitionLevelConsumer.java:89) ~[pinot-pulsar-0.13.0-SNAPSHOT-shaded.jar:0.13.0-SNAPSHOT-68b4096745272706af3e09f08ecbbd89eac7e3a3] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:427) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:648) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-0bbb9455eb7aa374e0c8b1e2c5ec0a4d6cd7a93e] Mar 29 08:50:57 pinot-hosts-9-server bash[2258505]: at java.lang.Thread.run(Unknown Source) [?:?] ``` ### Anything else? It seems quite rare to have this issue but some large topic (more than 16 millions entries in few hours) can have them repeatedly sometimes ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
