siddharthteotia opened a new pull request #5167: Fix selection early termination test URL: https://github.com/apache/incubator-pinot/pull/5167 We are seeing some failures in our automation jobs where Selection Early termination test is failing spuriously when verifying the numDocsScanned. The test assumes that each single thread would have done it's part of the work inside SelectionOnlyOperator.nextBlock() and returned from there. Take an example: LIMIT: 20480 threads per server: 5 num servers: 2 num segments (operators) per thread: 2 docs per segment: 30000 So as per the changes made in early termination PR, each thread will break out immediately after finishing the nextBlock() call on the first operator/segment. We check for numDocsScanned as limit * num servers * num threads per server => 20480 * 2 * 5 = 204800 Now consider the fact that 5th thread is a bit slow -- it is still executing the SelectionOnlyOperator.nextBlock() code and continuously updating it's local numDocsScanned The root thread in CombineOperator has already broken out because it has found a mergedBlock across all worker threads that has records >= LIMIT Next we compute total numDocsScanned across all operators (10) for statistics so it adds up something like this: 20480 + 20480 + 20480 + 20480 + X (X is slightly less than 20480) + 0 + 0 + 0 + 0 + 0 => something just under 204800 In the test we should assert for expected range of numDocsScanned considering the max would be when each of the 5 threads finished the first segment thus leading to a perfect total of 204800 Also added a TODO for a small optimization in CombineOperator
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
