comphead commented on code in PR #7662:
URL: https://github.com/apache/datafusion-comet/pull/7662#discussion_r4608074477
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala:
##########
@@ -136,16 +136,23 @@ case class CometIcebergNativeScanExec(
// tasks here would run a placeholder subquery. At execution,
CometNativeExec sizes the native
// RDD from perPartitionData instead.
//
- // Spark 3.4's BatchScanExec reports SinglePartition for one input
partition, which it counts
- // before any DPP filtering, and Spark plans no exchange above it on that
basis. The scan reports
- // the same. Otherwise EnsureRequirements, which AQE runs again over a DPP
subquery's converted
- // plan, inserts an exchange inside a native block that is already
serialized (#7422).
- // serializePartitions keeps that one partition when DPP prunes it, as Spark
does.
+ // Spark 3.4's BatchScanExec reports SinglePartition for one input
partition, and every supported
+ // version reports a key-grouped partitioning (KeyGroupedPartitioning,
called KeyedPartitioning
+ // from Spark 4.2) for a storage-partitioned scan. Spark takes both from the
partitions it plans,
+ // before any DPP filtering, and plans the operators above the scan on that
basis, with no
+ // exchange for a join or aggregate that the partitioning already satisfies.
The scan reports the
+ // same, so reading it resolves no DPP subquery. Otherwise
EnsureRequirements, which AQE runs
+ // again over a DPP subquery's converted plan, inserts exchanges inside a
native block that is
+ // already serialized (#7422, #7441). serializePartitions keeps one
partition for each of Spark's
+ // partitions when DPP prunes some, as Spark does.
override lazy val outputPartitioning: Partitioning =
- if (originalPlan != null && originalPlan.outputPartitioning ==
SinglePartition) {
- SinglePartition
- } else {
+ if (originalPlan == null) {
UnknownPartitioning(0)
+ } else {
+ originalPlan.outputPartitioning match {
+ case _: UnknownPartitioning => UnknownPartitioning(0)
+ case reported => reported
+ }
Review Comment:
When `BatchScanExec` reports no partitioning, it returns
`UnknownPartitioning(0)` from `SparkPlan.outputPartitioning` on every version I
checked (`v3.4.3`, `v3.5.8`, `v4.0.1`, `v4.1.1`, `v4.2.1`). So this match
always returns its input. `CometNativeScanExec.outputPartitioning` returns
`originalPlan.outputPartitioning` directly for a bucketed scan, and this could
do the same:
```suggestion
originalPlan.outputPartitioning
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]