rkrishn7 commented on issue #17171: URL: https://github.com/apache/datafusion/issues/17171#issuecomment-3282832934
I'm taking a stab at this in https://github.com/apache/datafusion/pull/17529! re: sharing the table - currently the build side tables are per partition and I'm not sure it would be that beneficial to try to use some sort of shared, partition-aware table. Even though it comes at the cost of extra memory, I think it's probably okay to copy build side hashes into a new allocation when constructing the dynamic filter, which the row filtering expression we push down ultimately uses. We can track this via the same memory consumer and it will be smaller than the collective size of the hash tables from all partitions (since we're not tracking row indexes). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
