GitHub user robo-todd added a comment to the discussion: Join problems with custom TableProviders
Adding some details: This is with DataFusion 48 and 49 versions. Also for a join of this size (64 rows approximately?) I would not expect to see more than one scan of each incoming table. The custom TableProviders fully obey the column projection for scan requests, but are otherwise pretty simple internally. They are returning complete single record batches of roughly 64 rows, so I'm mystified how join is producing streams of ultra-tiny 1 and 2 row record batches as output. GitHub link: https://github.com/apache/datafusion/discussions/16981#discussioncomment-13947413 ---- This is an automatically sent email for github@datafusion.apache.org. To unsubscribe, please send an email to: github-unsubscr...@datafusion.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
