comphead commented on code in PR #7662:
URL: https://github.com/apache/datafusion-comet/pull/7662#discussion_r4608074477


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometIcebergNativeScanExec.scala:
##########
@@ -136,16 +136,23 @@ case class CometIcebergNativeScanExec(
   // tasks here would run a placeholder subquery. At execution, 
CometNativeExec sizes the native
   // RDD from perPartitionData instead.
   //
-  // Spark 3.4's BatchScanExec reports SinglePartition for one input 
partition, which it counts
-  // before any DPP filtering, and Spark plans no exchange above it on that 
basis. The scan reports
-  // the same. Otherwise EnsureRequirements, which AQE runs again over a DPP 
subquery's converted
-  // plan, inserts an exchange inside a native block that is already 
serialized (#7422).
-  // serializePartitions keeps that one partition when DPP prunes it, as Spark 
does.
+  // Spark 3.4's BatchScanExec reports SinglePartition for one input 
partition, and every supported
+  // version reports a key-grouped partitioning (KeyGroupedPartitioning, 
called KeyedPartitioning
+  // from Spark 4.2) for a storage-partitioned scan. Spark takes both from the 
partitions it plans,
+  // before any DPP filtering, and plans the operators above the scan on that 
basis, with no
+  // exchange for a join or aggregate that the partitioning already satisfies. 
The scan reports the
+  // same, so reading it resolves no DPP subquery. Otherwise 
EnsureRequirements, which AQE runs
+  // again over a DPP subquery's converted plan, inserts exchanges inside a 
native block that is
+  // already serialized (#7422, #7441). serializePartitions keeps one 
partition for each of Spark's
+  // partitions when DPP prunes some, as Spark does.
   override lazy val outputPartitioning: Partitioning =
-    if (originalPlan != null && originalPlan.outputPartitioning == 
SinglePartition) {
-      SinglePartition
-    } else {
+    if (originalPlan == null) {
       UnknownPartitioning(0)
+    } else {
+      originalPlan.outputPartitioning match {
+        case _: UnknownPartitioning => UnknownPartitioning(0)
+        case reported => reported
+      }

Review Comment:
   When `BatchScanExec` reports no partitioning, it returns 
`UnknownPartitioning(0)` from `SparkPlan.outputPartitioning` on every version I 
checked (`v3.4.3`, `v3.5.8`, `v4.0.1`, `v4.1.1`, `v4.2.1`). So this match 
always returns its input. `CometNativeScanExec.outputPartitioning` returns 
`originalPlan.outputPartitioning` directly for a bucketed scan, and this could 
do the same:
   
   ```suggestion
         originalPlan.outputPartitioning
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to