andygrove commented on PR #2558: URL: https://github.com/apache/datafusion-ballista/pull/2558#issuecomment-6049101861
Thanks @villebro, good call. I filed apache/datafusion#26130 to prune files by statistics in `ListingTable` before it builds the file groups. DataFusion mostly hides this skew in a single process because idle partitions steal files from their siblings. We can't rely on that in Ballista since each partition runs as its own task. If the upstream change lands, DataFusion's existing file splitting would spread the remaining files and we could drop this rule. Until then this rule covers it on our side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
