Jackie-Jiang commented on issue #8931: URL: https://github.com/apache/pinot/issues/8931#issuecomment-1160781895
There are 2 part of the issue: 1. Partition pruning (`SinglePartitionColumnSegmentPruner`): we re-calculate the matching partition on a per segment per query basis, which can be very costly when there are lots of segments. Instead, we should be able to pre-group the segments to each partition id so that we only need to calculate the matching partition once 2. Time pruning (`TimeSegmentPruner`): This is where the interval tree is introduced. Currently we first get all the available segments matching the time range, then check if the segment is selected. Instead, we can probably only match the selected segments from the previous step. Some profile number can help identify the hotspot of the algorithm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
