richardstartin commented on PR #8879: URL: https://github.com/apache/pinot/pull/8879#issuecomment-1152554035
If you have dictionaries for the sort dimension, you could sometimes (whenever the sort key would fit into 64 bits) define a sort key the same way we define a group key in the group by. Then you could pass through the file once, extract the sort dimensions, compute the sort key, store it in an array, then sort that array, updating the sortedRowIds array as necessary. You could consider using radix sort here, which might be faster than quicksort. This way, you only read the file once during the sort, and read it sequentially, which will reduce the number of page faults drastically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
