akshaytayal commented on code in PR #13519:
URL: https://github.com/apache/gluten/pull/13519#discussion_r4507455109


##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
     }
   }
 
+  private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+    override def computeValue(clazz: Class[_]): Method = 
clazz.getMethod("shuffleBlockResolver")

Review Comment:
   `shuffleBlockResolver` is a public member of Spark 4.3's 
`BlockingShuffleManager` (`private[spark]` is public in bytecode), so 
`getMethod` finds it on every implementing class, including inherited ones.
   
   Posted by an AI assistant. Flag anything wrong here.



##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
     }
   }
 
+  private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+    override def computeValue(clazz: Class[_]): Method = 
clazz.getMethod("shuffleBlockResolver")
+  }
+
+  /**
+   * Returns the block resolver of a registered shuffle manager. Spark 4.3 
declares
+   * `shuffleBlockResolver` only on `BlockingShuffleManager` (which doesn't 
exist before 4.3), so it
+   * is looked up reflectively to compile against every supported Spark 
version.
+   */
+  private def shuffleBlockResolverOf(manager: ShuffleManager): 
ShuffleBlockResolver = {
+    shuffleBlockResolverAccessor
+      .get(manager.getClass)
+      .invoke(manager)
+      .asInstanceOf[ShuffleBlockResolver]
+  }

Review Comment:
   `shuffleBlockResolver` is a public member of Spark 4.3's 
`BlockingShuffleManager` (`private[spark]` is public in bytecode), so 
`getMethod` finds it on every implementing class, including inherited ones.
   
   Posted by an AI assistant. Flag anything wrong here.



##########
gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/GlutenCostEvaluator.scala:
##########
@@ -33,8 +33,20 @@ case class GlutenCostEvaluator() extends CostEvaluator with 
SQLConfHelper {
     val forceOptimizeSkewedJoin =
       
conf.getConfString("spark.sql.adaptive.forceOptimizeSkewedJoin").toBoolean
     val clazz = 
Utils.classForName("org.apache.spark.sql.execution.adaptive.SimpleCostEvaluator")
-    val ctor = clazz.getConstructor(classOf[Boolean])
-    
ctor.newInstance(forceOptimizeSkewedJoin.asInstanceOf[Object]).asInstanceOf[CostEvaluator]
+    if (SparkVersionUtil.gteSpark43) {
+      // Spark 4.3 added the `countLocalSort` parameter.
+      val countLocalSort =
+        
conf.getConfString("spark.sql.adaptive.costEvaluator.countLocalSort.enabled").toBoolean

Review Comment:
   `spark.sql.adaptive.costEvaluator.countLocalSort.enabled` is a registered 
SQLConf entry in Spark 4.3, and this branch only runs on Spark 4.3+, so 
`getConfString` returns its default when the key is unset.
   
   Posted by an AI assistant. Flag anything wrong here.



##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
     val executorCores = SparkResourceUtil.getExecutorCores(conf)
     // spark.task.cpus is read raw here, which bypasses Spark's own 
checkValue(_ > 0) (and on
     // Spark < 4.2 that positivity check does not exist at all). getTaskSlots 
runs during driver
-    // plugin init, before Spark validates the value, so fail fast on a 
non-positive setting rather
-    // than dividing by it: a zero would throw an opaque "/ by zero" 
ArithmeticException and a
-    // negative would silently produce negative task slots and off-heap 
budgets.
-    val taskCores = conf.getInt("spark.task.cpus", 1)
-    require(taskCores > 0, s"spark.task.cpus should be positive, but was 
$taskCores")
+    // plugin init, before Spark validates the value. Spark 4.3 allows 
fractional task cpus
+    // (SPARK-58192), so read it as a decimal.
+    getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus", 
"1").trim))

Review Comment:
   A malformed `spark.task.cpus` is a fatal configuration error either way, and 
the stack trace points at `SparkResourceUtil`, so no extra handling is added 
here.
   
   Posted by an AI assistant. Flag anything wrong here.



##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
     val executorCores = SparkResourceUtil.getExecutorCores(conf)
     // spark.task.cpus is read raw here, which bypasses Spark's own 
checkValue(_ > 0) (and on
     // Spark < 4.2 that positivity check does not exist at all). getTaskSlots 
runs during driver
-    // plugin init, before Spark validates the value, so fail fast on a 
non-positive setting rather
-    // than dividing by it: a zero would throw an opaque "/ by zero" 
ArithmeticException and a
-    // negative would silently produce negative task slots and off-heap 
budgets.
-    val taskCores = conf.getInt("spark.task.cpus", 1)
-    require(taskCores > 0, s"spark.task.cpus should be positive, but was 
$taskCores")
+    // plugin init, before Spark validates the value. Spark 4.3 allows 
fractional task cpus
+    // (SPARK-58192), so read it as a decimal.
+    getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus", 
"1").trim))
+  }
+
+  /** Returns floor(executorCores / taskCpus), at least one, like Spark's own 
slot computation. */
+  def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {

Review Comment:
   A malformed `spark.task.cpus` is a fatal configuration error either way, and 
the stack trace points at `SparkResourceUtil`, so no extra handling is added 
here.
   
   Posted by an AI assistant. Flag anything wrong here.



##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
     val executorCores = SparkResourceUtil.getExecutorCores(conf)
     // spark.task.cpus is read raw here, which bypasses Spark's own 
checkValue(_ > 0) (and on
     // Spark < 4.2 that positivity check does not exist at all). getTaskSlots 
runs during driver
-    // plugin init, before Spark validates the value, so fail fast on a 
non-positive setting rather
-    // than dividing by it: a zero would throw an opaque "/ by zero" 
ArithmeticException and a
-    // negative would silently produce negative task slots and off-heap 
budgets.
-    val taskCores = conf.getInt("spark.task.cpus", 1)
-    require(taskCores > 0, s"spark.task.cpus should be positive, but was 
$taskCores")
+    // plugin init, before Spark validates the value. Spark 4.3 allows 
fractional task cpus
+    // (SPARK-58192), so read it as a decimal.
+    getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus", 
"1").trim))
+  }
+
+  /** Returns floor(executorCores / taskCpus), at least one, like Spark's own 
slot computation. */
+  def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {
+    // Fail fast on a non-positive setting rather than dividing by it: a zero 
would throw an opaque
+    // ArithmeticException and a negative would silently produce negative task 
slots and off-heap
+    // budgets.
+    require(taskCpus > 0, s"spark.task.cpus should be positive, but was 
$taskCpus")

Review Comment:
   A malformed `spark.task.cpus` is a fatal configuration error either way, and 
the stack trace points at `SparkResourceUtil`, so no extra handling is added 
here.
   
   Posted by an AI assistant. Flag anything wrong here.



##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,18 @@ object Validators {
     }
   }
 
+  private object FallbackByUnsupportedSparkFeatures extends Validator {
+    override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+      // The scan reports `SinglePartition` and Spark may have elided a 
shuffle above it, while the
+      // native scan doesn't coalesce its splits into a single task.
+      case p: FileSourceScanExec if 
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>
+        fail(s"${p.nodeName} uses single-task execution, which is not 
supported")
+      case p: LocalTableScanExec if p.outputPartitioning.numPartitions == 1 =>
+        fail(s"${p.nodeName} uses single-task execution, which is not 
supported")
+      case _ => pass()
+    }
+  }

Review Comment:
   The Spark 4.3 `MarkSingleTaskExecutionSuite` runs under Gluten and executes 
single-task file scans end to end. The `LocalTableScanExec` case applies only 
when `spark.gluten.sql.columnar.localTableScan` is enabled (off by default).
   
   Posted by an AI assistant. Flag anything wrong here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to