michalcukierman opened a new issue, #21074: URL: https://github.com/apache/pulsar/issues/21074
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version 3.1.0 using official Helm chart Latest build from master using official Helm chart ### Minimal reproduce step Use 3 bookies, 3 brokers, 3 proxies 1. Create compacted, partitioned topic with the compaction threshold set to 1GB 2. Produce 100k of messages of 100 kb size 3. Create exclusive consumer and start reading 4. Produce 100k of messages with the same keys -- wait for the messages to be produced and the compaction to finish 6. Create new exclusive consumer and read all the messages ### What did you expect to see? Compaction finish successfully. First consumer receives between 100k - 200k messages Second consumer receives 100k messages ### What did you see instead? - I see in the error logs that the compaction fails. <img width="1618" alt="Screenshot 2023-08-27 at 18 02 33" src="https://github.com/apache/pulsar/assets/4356553/b26ff482-de67-4b6c-96d9-6cec98bc8e13";> - The first consumer often starts to receive messages, that were already delivered (millions of messages, It never ends) - The second consumer cannot receive all the messages. Sometimes it is able to finish, sometimes is not (falls into the loop) - The backlog of the subscription does not change while the consumers are reading Reads comes from one broker, the backlog is not changing (the screenshot is from my environment/system): <img width="1679" alt="Screenshot 2023-08-27 at 18 29 22" src="https://github.com/apache/pulsar/assets/4356553/87803ebd-3488-4092-bb64-15444123c21b";> ### Anything else? The reproduction steps depends on the used setup, speed of the persistent storage. It's possible that the re balancing or adding new broker affects the steps. I cannot provide reproducible way, as it happens randomly. Nevertheless it happens to us almost every day. My internal stats: [internal.stats.txt](https://github.com/apache/pulsar/files/12448404/internal.stats.txt) ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
