tschwarzinger commented on PR #24921: URL: https://github.com/apache/datafusion/pull/24921#issuecomment-6574216925
I think these two are definitely related and all the PRs around Parquet I/O are very interesting to me. But from what I can tell they will not go beyond a single scan right? So, for example, if we have 5 small and 2 big scans in a plan, the 5 small scans will still get executed serially when using hash joins. This PR should enable fetching the 5 small scans in parallel, then do the two large scans sequentially with dynamic filters available. It's a quite easy thing to add downstream and I think up streaming may only makes sense if a broader set of users need it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
