gene-bordegaray commented on issue #24438: URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5482584384
Thank you for the in depth explanation and discussion. I caight up on this and review the first testing PR. Thank you @xavlee A trend I am seeing across the project is the need for more partitioning properties to properly be represented and taken avantage of. This largely started with preserving pre partitioned data to eliminate shuffles, then new range partitioning use cases, and more like this are popping up in the repo. Given us at Datadog are one of the main drivers of these efforts, it seems that others using Datafusion at scale also see the use case for this. I think it would be ineresting to start a discussion about where the community wants to see partitioning head and how DF should expose properties that ir can take advantage of. Hopefully others here agree with that and I can start this discussion and link relevant work going on around this. 🙇 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
