michalcukierman opened a new issue, #20968: URL: https://github.com/apache/pulsar/issues/20968
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version - Affected Pulsar versions: - 3.0.1 - 3.0.0 - Tested environments - GCP Kubernentes Cluster - 3-3-3-4 (Zk-proxy-broker-BK), Qurum 2-2-2 - MicroK8S local cluster - 1-1-3-1 (Zk-proxy-broker-BK), Qurum 1-1-1 Deplyments done using official Helm Chart. ### Minimal reproduce step Checkout: https://github.com/websight-io/pulsar-chaos-test/tree/main/environment-local and follow README.MD OR: 1. Create partitioned (12 partitions), set retention to -1, -1 2. Create a producer and start writing to a topic 3. Create a consumer with a new subscription set to `Earliest` position and start reading the messages 4. Restart all Broker Pods one-by-one 5. Wait for the brokers to start (2-3 minutes) 6. Stop producer created in step 2 7. Stop consumer created in step 3 8. Create a new consumer with a new subscription set to `Earliest` position and read all the messages ### What did you expect to see? Both consumers read the same number of messages. Topic does not contain all the messages that were stored in it. ### What did you see instead? Consumer created after the restart cannot read the same number of messages as the one created before the restart. Topic should contain all the messages that were stored in it. ### Anything else? - Retention is set before starting the set. It's preserved and can be verified: `bin/pulsar-admin topics get-retention persistent://websight/dxp/chaos-test` ``` { "retentionTimeInMinutes" : -1, "retentionSizeInMB" : -1 } ``` - Pulsar Manager shows that some of the partitions are empty after restart (or the data is not equaly distributed) <img width="1217" alt="Screenshot 2023-08-09 at 18 40 31" src="https://github.com/apache/pulsar/assets/4356553/e0326f27-0031-4dcd-95e3-98968cda8962";> - Pulsar Manager state from before the restart: <img width="1218" alt="Screenshot 2023-08-09 at 18 41 28" src="https://github.com/apache/pulsar/assets/4356553/aaa45b43-03e7-4f48-a994-889a466d80eb";> ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
