akshaytayal commented on code in PR #13519:
URL: https://github.com/apache/gluten/pull/13519#discussion_r4648721263


##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,18 @@ object Validators {
     }
   }
 
+  private object FallbackByUnsupportedSparkFeatures extends Validator {
+    override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+      // The scan reports `SinglePartition` and Spark may have elided a 
shuffle above it, while the
+      // native scan doesn't coalesce its splits into a single task.
+      case p: FileSourceScanExec if 
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>
+        fail(s"${p.nodeName} uses single-task execution, which is not 
supported")
+      case p: LocalTableScanExec if p.outputPartitioning.numPartitions == 1 =>
+        fail(s"${p.nodeName} uses single-task execution, which is not 
supported")
+      case _ => pass()
+    }
+  }

Review Comment:
   Update: 7ff7c67 removes the `LocalTableScanExec` part of this rule. 
`VeloxLocalTableScanTransformer` now produces the single partition it reports 
(22de284 in #13518), so that scan stays native. The `FileSourceScanExec` part 
is unchanged and is covered by `GlutenMarkSingleTaskExecutionSuite` (enabled in 
#13520).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to