peter-toth commented on code in PR #59789:
URL: https://github.com/apache/spark/pull/59789#discussion_r4441657670
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:
##########
@@ -389,6 +389,128 @@ trait KeyGroupedPartitioningRuntimeFilterTests extends
KeyGroupedPartitioningSui
checkAnswer(df, Seq(Row(1L, 42.0f), Row(1L, 44.0f)))
}
}
+
+ test("SPARK-60260: a storage-partitioned join keeps the planned keyed layout
when v2 bucketing " +
+ "is disabled after planning") {
+ // reportedKeyedPartitioning used to read
spark.sql.sources.v2.bucketing.enabled at execution,
+ // so turning it off between planning and execution dropped the keyed
layout on the scan that
+ // PlanAdaptiveDynamicPruningFilters rebuilds at stage creation, and the
sort-merge join then
+ // lined partitions up wrong. The decision is frozen at planning now.
Covered across three
+ // shapes, each planned with bucketing on and then executed with it off,
AQE on.
Review Comment:
**Finding 15.** Shape 4 runs a scalar-subquery runtime filter with DPP off,
and with AQE off too. The header comment still says three shapes, AQE on:
```suggestion
// lined partitions up wrong. The decision is frozen at planning now.
Covered across four
// shapes, each planned with bucketing on and then executed with it off.
Shapes 1-3 use DPP
// with AQE on, and shape 4 a scalar subquery with DPP off and AQE on
and off.
```
Three lines in `check` say DPP where shape 4 has a scalar subquery:
- `:402`: "the keyed layout and DPP are decided here" could say "the runtime
filters".
- `:413`: "expected dynamic partition pruning on the keyed scan" could say
"a runtime filter".
- `:425`: "The scan with the DPP filter is the one
`PlanAdaptiveDynamicPruningFilters` rebuilds" holds only for shapes 1-3.
The description's "DPP on the keyed scan" has the same gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
