shenyu0127 commented on issue #11084: URL: https://github.com/apache/pinot/issues/11084#issuecomment-1633416641
@Jackie-Jiang helped me identify the root cause. The realtime segments are created non-deterministically because the Kafka partitioning is using the system time ([ref](https://github.com/apache/pinot/blob/8171cfb000e68c40f08bd69a9c1f84e99b7c38c4/pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/ClusterIntegrationTestUtils.java#L366)). There is a small chance some segments do not have the `ActualElapsedTime = -9999` record and gets pruned, so the `numEntriesScannedInFilter` is smaller than `numTotalDocs`, and the assert fails. I am able to reproduce the bug by playing with the Kafka partition key generation: totalDocs = 230546, numEntriesScannedInFilter = 230545  This is because we have a consuming segment that has only 1 record which is not `ActualElapsedTime = -9999` and gets pruned.   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
