vrajat commented on PR #12608: URL: https://github.com/apache/pinot/pull/12608#issuecomment-1986390389
I tested this commit manually. Gist of quickstart: https://gist.github.com/vrajat/5db2715535db325dde9af8e4a029da47 The workflow is as follows: * Insert a batch of 400 messages. * Pinot ingests these messages * Pause consumption * Insert another batch of 400 messages * Delete records up to offset 500. So 100 records are missing. * Resume consumption. O/p ``` ***** Cluster is running ***** ***** Publish first batch of data to kafka streams ***** ***** Starting githubEvents data stream and publishing to Kafka ***** ***** Pausing consumption ***** ***** Publish second batch of data to kafka streams ***** ***** Starting githubEvents data stream and publishing to Kafka ***** Executing records delete operation Records delete operation completed: partition: githubEvents-1 low_watermark: 500 partition: githubEvents-0 low_watermark: 500 Offset is 500 Offset is 500 ***** Publish third batch of data to kafka streams ***** ***** Starting githubEvents data stream and publishing to Kafka ***** ***** Resume consumption ***** You can always go to http://localhost:9000 to play around in the query console 2024/03/08 12:38:01.004 ERROR [PinotLLCRealtimeSegmentManager] [pool-23-thread-2] Data lost from offset: 400 to: 500 for partition: 0 of table: githubEvents_REALTIME 2024/03/08 12:38:01.013 ERROR [PinotLLCRealtimeSegmentManager] [pool-23-thread-2] Data lost from offset: 400 to: 500 for partition: 1 of table: githubEvents_REALTIME ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
