cetra3 commented on code in PR #27319:
URL: https://github.com/apache/datafusion/pull/27319#discussion_r4581073833
##########
datafusion/physical-expr/src/equivalence/class.rs:
##########
@@ -699,6 +699,27 @@ impl EquivalenceGroup {
.collect()
}
+ /// Projects `expr` through the mapping, or else projects another member of
+ /// its equivalence class.
+ ///
+ /// The members of a class are equal on every row, so a projected member
+ /// describes the same partitioning or ordering as `expr`. For example, if
+ /// `x` is in a class with `get_field(s, 'k')`, a projection that keeps `s`
+ /// but drops `x` projects `x` to `get_field(s, 'k')`. Literal members are
+ /// skipped, because a literal does not describe how rows are distributed.
+ fn project_expr_or_equivalent(
+ &self,
+ aug_mapping: &AugmentedMapping,
+ expr: &Arc<dyn PhysicalExpr>,
+ ) -> Option<Arc<dyn PhysicalExpr>> {
+ Self::project_expr_indirect(aug_mapping, expr).or_else(|| {
Review Comment:
Thanks, confirmed: `s['k'] || 'z'` plans without this PR and failed with it.
I took your patch in 49e86add18 with one change.
With `tried` as a list, a key that uses the dropped column twice still
failed, because the second use was treated as a cycle:
```sql
SELECT s['k'] || s['k'] FROM mark_join
WHERE s['k'] = 'a' OR s['k'] || s['k'] IN (SELECT s['k'] || s['k'] FROM
mark_join)
GROUP BY 1;
-- SanityCheckPlan error
```
`tried` is now a map from each expression to its result. An entry of `None`
while the expression is in progress stops cycles, and the stored result serves
later uses of the same expression. I added that query to the slt file and a
unit test for it (`test_project_dropped_child_used_twice`), along with your slt
case and your two unit tests.
##########
datafusion/sqllogictest/test_files/equivalence_properties.slt:
##########
@@ -375,6 +375,60 @@ physical_plan
05)--------RepartitionExec: partitioning=RoundRobinBatch(2),
input_partitions=1, maintains_sort_order=true
06)----------StreamingTableExec: partition_sizes=1, projection=[a, b, c],
infinite_source=true, output_ordering=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST,
c@2 ASC NULLS LAST]
+# A partitioned mark join on a struct field, under an aggregate grouped by the
+# same field. The join output is already hash partitioned on the field, so the
+# aggregate needs no repartition. Projection pushdown then moves the field
+# access into the scan and drops the join key from the join output. The join
+# must still know its partitioning through the equivalent field access, or the
+# plan fails the sanity check.
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+set datafusion.optimizer.hash_join_single_partition_threshold_rows = 0;
+
+statement ok
+COPY (SELECT named_struct('k', 'a') AS s)
+TO 'test_files/scratch/equivalence_properties/mark_join/1.parquet';
+
+statement ok
+COPY (SELECT named_struct('k', 'b') AS s)
+TO 'test_files/scratch/equivalence_properties/mark_join/2.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE mark_join STORED AS PARQUET
+LOCATION 'test_files/scratch/equivalence_properties/mark_join/';
+
+query T rowsort
+SELECT s['k'] FROM mark_join
+WHERE s['k'] = 'a' OR s['k'] IN (SELECT s['k'] FROM mark_join)
+GROUP BY 1;
+----
+a
+b
+
+query TT
+EXPLAIN SELECT s['k'] FROM mark_join
+WHERE s['k'] = 'a' OR s['k'] IN (SELECT s['k'] FROM mark_join)
+GROUP BY 1;
+----
+physical_plan
+01)ProjectionExec: expr=[__datafusion_extracted_2@0 as mark_join.s[k]]
+02)--AggregateExec: mode=SinglePartitioned, gby=[__datafusion_extracted_2@0 as
__datafusion_extracted_2], aggr=[]
+03)----FilterExec: __datafusion_extracted_3@1 = a OR mark@0,
projection=[__datafusion_extracted_2@2]
+04)------ProjectionExec: expr=[mark@0 as mark, get_field(s@1, k) as
__datafusion_extracted_3, get_field(s@1, k) as __datafusion_extracted_2]
+05)--------HashJoinExec: mode=Partitioned, join_type=LeftMark,
on=[(__datafusion_extracted_4@0, mark_join.s[k]@0)], projection=[mark@2, s@1]
+06)----------RepartitionExec: partitioning=Hash([__datafusion_extracted_4@0],
2), input_partitions=2
+07)------------DataSourceExec: file_groups={2 groups:
[[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/1.parquet],
[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/2.parquet]]},
projection=[get_field(s@0, k) as __datafusion_extracted_4, s],
file_type=parquet
+08)----------RepartitionExec: partitioning=Hash([mark_join.s[k]@0], 2),
input_partitions=2
+09)------------DataSourceExec: file_groups={2 groups:
[[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/1.parquet],
[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/equivalence_properties/mark_join/2.parquet]]},
projection=[get_field(s@0, k) as mark_join.s[k]], file_type=parquet,
predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
+
+statement ok
Review Comment:
Added in 49e86add18. I confirmed it fails with the `InterleaveExec` error
without the fix and returns `a`, `b` with it.
##########
datafusion/physical-expr/src/equivalence/class.rs:
##########
@@ -699,6 +699,27 @@ impl EquivalenceGroup {
.collect()
}
+ /// Projects `expr` through the mapping, or else projects another member of
+ /// its equivalence class.
+ ///
+ /// The members of a class are equal on every row, so a projected member
+ /// describes the same partitioning or ordering as `expr`. For example, if
+ /// `x` is in a class with `get_field(s, 'k')`, a projection that keeps `s`
+ /// but drops `x` projects `x` to `get_field(s, 'k')`. Literal members are
+ /// skipped, because a literal does not describe how rows are distributed.
+ fn project_expr_or_equivalent(
+ &self,
+ aug_mapping: &AugmentedMapping,
+ expr: &Arc<dyn PhysicalExpr>,
+ ) -> Option<Arc<dyn PhysicalExpr>> {
+ Self::project_expr_indirect(aug_mapping, expr).or_else(|| {
+ self.get_equivalence_class(expr)?
+ .iter()
+ .filter(|member| *member != expr && !member.is::<Literal>())
Review Comment:
Agreed, I dropped the skip and its doc sentence in 49e86add18. The tests
still pass without it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]