kishoreg commented on PR #8879: URL: https://github.com/apache/pinot/pull/8879#issuecomment-1152868478
Another idea along the lines of what @richardstartin mentioned but it does not have to be dictionary encoded and potentially avoid the need to sort the file on disk and sort complexity reduced from a function of num_rows to cardinality of the sort column In the first pass, create a map of <key, bitmap> sort this map/array on key. you can now create a sorted doc id array (this is also optional) you can simply create a record reader and pass the hashmap that will dictate the reading order. We do something like this in real-time segment converter when we flush real-time to disk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
