alamb commented on issue #15191: URL: https://github.com/apache/datafusion/issues/15191#issuecomment-5258464010
> ProgressiveEval streams and reduce the amount of merging required. Ideally we could have access to the sort-column values from the first and last rows though. Yeah, this is a modification of the algorithm The benefit of progressive eval as written is that it doesn't even open the files (no IO at all) for later on in the sort order -- if you need the first / last sort rows you either have to open the file and read them (which can be expensive if they are on object store, for example) or cache them somewhere 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
