jpatidar30 opened a new issue, #2715: URL: https://github.com/apache/iggy/issues/2715
Description I am encountering a critical issue where a consumer unexpectedly jumps from an intermediate offset directly to the end of the stream (the latest offset). This behavior is non-deterministic and occurs after ingesting a large volume of data (~50M records). Behavior Observed The Jump: While consuming, the fetch logic might work perfectly up to a certain point (e.g., offset 5,000,000). On the next poll, instead of receiving 5,000,001, the consumer is suddenly positioned at the end of the stream (50,000,000), effectively skipping 45M records. Non-Determinism: The "jump point" is not fixed. A new consumer group/ID might jump at a completely different offset (e.g., at 12M instead of 5M). Retrying the same consumer sometimes works after 2 retries, and sometimes requires 10+ retries to successfully pass the "glitchy" offset. Data Integrity: The data does exist in the stream, as repeated retries eventually allow the consumer to read through the records that were previously skipped. Environment Information Iggy Version: Docker image with latest tag Storage: EBS GP3 (AWS) SDK Used: Rust -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
