rdtr commented on code in PR #13518:
URL: https://github.com/apache/gluten/pull/13518#discussion_r4572586005


##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,16 @@ object Validators {
     }
   }
 
+  private object FallbackByUnsupportedSparkFeatures extends Validator {
+    override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+      // The scan reports `SinglePartition` and Spark may have elided a 
shuffle above it, while the
+      // native scan doesn't coalesce its splits into a single task.
+      case p: FileSourceScanExec if 
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>

Review Comment:
   Doesn't `LocalTableScanExec` have the same single-task gap?
   
   In Spark 4.3 it also reports `SinglePartition` when marked 
(`LocalTableScanExec.scala:79-85`) and creates one empty partition for empty 
input (`:62-65`). `VeloxLocalTableScanTransformer` keeps 
`plan.outputPartitioning` (`VeloxLocalTableScanTransformer.scala:152`) but 
parallelizes into several slices (`:124`) and returns `emptyRDD` for empty 
input (`:114`).
   
   With `spark.gluten.sql.columnar.localTableScan=true` and single-task 
execution enabled, a global aggregation with the shuffle elided could return 
several rows, or none on empty input. Both configs are off by default, which is 
why "empty local relation + global aggregation returns one row" passes in 
`GlutenMarkSingleTaskExecutionSuite` in #13520.
   
   A follow-up could handle it in the transformer rather than with a fallback: 
when `outputPartitioning` is `SinglePartition`, parallelize into one slice and 
use `parallelize(Seq.empty, 1)` for empty input, as `LocalTableScanExec` does 
(`:59-72`). That keeps the scan native, and checking `outputPartitioning` needs 
no shim, since it is always `UnknownPartitioning(0)` before 4.3.



##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
     }
   }
 
+  private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+    override def computeValue(clazz: Class[_]): Method = 
clazz.getMethod("shuffleBlockResolver")

Review Comment:
   Nit: `clazz.getMethod("shuffleBlockResolver")` returns the implementing 
class's own method, and `invoke` throws `IllegalAccessException` if that class 
isn't public (a Java anonymous class, for example). Looking the method up once 
on the declaring type (`BlockingShuffleManager` on 4.3, `ShuffleManager` 
before) avoids that and drops the `ClassValue`. Fine to leave as is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to