gabotechs commented on PR #18919: URL: https://github.com/apache/datafusion/pull/18919#issuecomment-3595267052
> Then submit follow up issues to address grouping files into partitions to help with higher cardinality. I just do not want to introduce to many things in this PR and adding this seems like another substantial PR in itself. I think @fmonjalet's suggestion (and mine) is to not introduce `Partitioning::KeyPartitioned` at all and just enhance the existing `Partitioning::Hash` with smarter capabilities, so it's more a matter of reducing the amount of things this PR already has rather than adding on top. I think avoiding extra repartitions by reusing an existing hash partitioning in an operator that requires some other hash partitioning that just partially matches the incoming one should be achievable without introducing new partitioning methods. Also note that ideally the amount of partitions in a data stream should not be given by the nature of the data, but by the amount of CPUs a machine has, that's what allows us to optimize for resource usage regardless of how the data happened to be laid out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
