mithuncy opened a new issue, #23572: URL: https://github.com/apache/datafusion/issues/23572
Related: - EPIC: #22395 - Windows: #23289 (PR #23416, where this follow-up was requested) - Aggregations: #23239 - Joins: #23184 - PartitionedTopK: #23290 (PR #23355) ## Is your feature request related to a problem or challenge? Each operator that opts into range satisfaction currently re-tests the same shared config machinery (`datafusion.optimizer.subset_repartition_threshold`, `datafusion.optimizer.preserve_file_partitions`, `datafusion.execution.target_partitions`) through per-operator blocks in `range_partitioning.slt`. The machinery is shared, so the file grows the same permutations for every operator. This was raised in the #23416 review ([discussion](https://github.com/apache/datafusion/pull/23416#discussion_r3564136172)): keep `range_partitioning.slt` lean per operator, and cover the config permutations once, colocated with the decision logic. ## Describe the solution you'd like A table-driven test in `datafusion/core/tests/physical_optimizer/enforce_distribution.rs` asserting reuse (no `RepartitionExec` inserted; the existing Range partitioning satisfies the requirement) vs. hash repartition (`RepartitionExec: Hash` inserted) across combinations of: - key match: exact / subset / incompatible - `subset_repartition_threshold`: met / not met - `preserve_file_partitions`: met / not met - `target_partitions`: equal to / greater than input partition count Behavior verified while developing #23416 that the matrix must pin: - exact key, threshold not met, equal partition count → reuse (repartitioning gains no parallelism) - exact key, threshold not met, target > count, preserve not met → hash repartition - exact key, threshold not met, target > count, preserve met → reuse (the threshold and preserve conditions are OR'd in `ensure_distribution`) - exact key, threshold met, preserve not met, target > count → reuse - subset key, threshold not met, preserve not met → hash repartition - incompatible key → hash repartition in all configurations ## Describe alternatives you've considered None. ## Additional context - Once the matrix lands, the equivalent per-operator slt permutations in the aggregate section of `range_partitioning.slt` can also be slimmed, and future operator opt-ins only need operator-specific slt cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
