ShayanGho commented on code in PR #25304:
URL: https://github.com/apache/datafusion/pull/25304#discussion_r4078666493
##########
datafusion/physical-optimizer/src/join_selection.rs:
##########
@@ -196,6 +201,10 @@ pub(crate) fn try_collect_left(
ignore_threshold: bool,
context: &dyn PhysicalOptimizerContext,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+ if hash_join.inputs_satisfy_partitioned_requirements()? {
+ return Ok(None);
+ }
+
Review Comment:
Thanks, @stuhood. I tested the Hive-partitioned case and found a
wrong-results regression with preserve_file_partitions=1.
##########
datafusion/physical-optimizer/src/join_selection.rs:
##########
@@ -196,6 +201,10 @@ pub(crate) fn try_collect_left(
ignore_threshold: bool,
context: &dyn PhysicalOptimizerContext,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+ if hash_join.inputs_satisfy_partitioned_requirements()? {
+ return Ok(None);
+ }
+
Review Comment:
This can drop rows with preserve_file_partitions=1: Hive scans advertise
Hash([k], n), but different value sets can place matching keys in different
partitions ([#23436](https://github.com/apache/datafusion/issues/23436)).
With this branch and target_partitions=3:
- Dim {A,B,C,D} → [[A,D],[B],[C]]
- Fact {B,C,D} → [[B],[C],[D]]
Joining on k returns 0 rows, versus 3 with this hunk reverted. The
underlying bug already affects non-broadcast joins; this PR extends it to small
tables previously protected by CollectLeft.
Dynamic-filter pushdown already [rejects Hash/Hash under this
setting](https://github.com/apache/datafusion/blob/47353981c045415ba28a382976340c894d389bd8/datafusion/physical-plan/src/joins/hash_join/exec.rs#L1005),
which also explains the lost filters in the updated plans. Could we apply the
same guard here and add this regression test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
