Copilot commented on code in PR #13519:
URL: https://github.com/apache/gluten/pull/13519#discussion_r4505826284


##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
     }
   }
 
+  private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+    override def computeValue(clazz: Class[_]): Method = 
clazz.getMethod("shuffleBlockResolver")

Review Comment:
   Using `Class#getMethod` only finds public methods. If Spark 4.3 exposes 
`shuffleBlockResolver` on `BlockingShuffleManager` with non-public visibility 
(or it’s inherited but not public), this will throw at runtime 
(NoSuchMethodException / IllegalAccessException). To make this robust across 
Spark builds, prefer `getDeclaredMethod` with `setAccessible(true)` (and/or try 
`getMethod` then fall back to `getDeclaredMethod`) and wrap failures with a 
clear error message that includes the shuffle manager class name.



##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
     }
   }
 
+  private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+    override def computeValue(clazz: Class[_]): Method = 
clazz.getMethod("shuffleBlockResolver")
+  }
+
+  /**
+   * Returns the block resolver of a registered shuffle manager. Spark 4.3 
declares
+   * `shuffleBlockResolver` only on `BlockingShuffleManager` (which doesn't 
exist before 4.3), so it
+   * is looked up reflectively to compile against every supported Spark 
version.
+   */
+  private def shuffleBlockResolverOf(manager: ShuffleManager): 
ShuffleBlockResolver = {
+    shuffleBlockResolverAccessor
+      .get(manager.getClass)
+      .invoke(manager)
+      .asInstanceOf[ShuffleBlockResolver]
+  }

Review Comment:
   Using `Class#getMethod` only finds public methods. If Spark 4.3 exposes 
`shuffleBlockResolver` on `BlockingShuffleManager` with non-public visibility 
(or it’s inherited but not public), this will throw at runtime 
(NoSuchMethodException / IllegalAccessException). To make this robust across 
Spark builds, prefer `getDeclaredMethod` with `setAccessible(true)` (and/or try 
`getMethod` then fall back to `getDeclaredMethod`) and wrap failures with a 
clear error message that includes the shuffle manager class name.



##########
gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/GlutenCostEvaluator.scala:
##########
@@ -33,8 +33,20 @@ case class GlutenCostEvaluator() extends CostEvaluator with 
SQLConfHelper {
     val forceOptimizeSkewedJoin =
       
conf.getConfString("spark.sql.adaptive.forceOptimizeSkewedJoin").toBoolean
     val clazz = 
Utils.classForName("org.apache.spark.sql.execution.adaptive.SimpleCostEvaluator")
-    val ctor = clazz.getConstructor(classOf[Boolean])
-    
ctor.newInstance(forceOptimizeSkewedJoin.asInstanceOf[Object]).asInstanceOf[CostEvaluator]
+    if (SparkVersionUtil.gteSpark43) {
+      // Spark 4.3 added the `countLocalSort` parameter.
+      val countLocalSort =
+        
conf.getConfString("spark.sql.adaptive.costEvaluator.countLocalSort.enabled").toBoolean

Review Comment:
   `getConfString(key)` will throw if the key is absent, which can break 
initialization across Spark patch versions or custom builds where this config 
is missing/renamed. Prefer using a defaulted lookup (where available) or 
guarding with `SparkConf.contains`/`SQLConf.contains` and defaulting to `false` 
to keep the evaluator resilient.



##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
     val executorCores = SparkResourceUtil.getExecutorCores(conf)
     // spark.task.cpus is read raw here, which bypasses Spark's own 
checkValue(_ > 0) (and on
     // Spark < 4.2 that positivity check does not exist at all). getTaskSlots 
runs during driver
-    // plugin init, before Spark validates the value, so fail fast on a 
non-positive setting rather
-    // than dividing by it: a zero would throw an opaque "/ by zero" 
ArithmeticException and a
-    // negative would silently produce negative task slots and off-heap 
budgets.
-    val taskCores = conf.getInt("spark.task.cpus", 1)
-    require(taskCores > 0, s"spark.task.cpus should be positive, but was 
$taskCores")
+    // plugin init, before Spark validates the value. Spark 4.3 allows 
fractional task cpus
+    // (SPARK-58192), so read it as a decimal.
+    getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus", 
"1").trim))

Review Comment:
   `BigDecimal(conf.get(...))` will throw a NumberFormatException for malformed 
values (e.g., empty string, non-numeric), which will surface during plugin init 
with a low-signal stack trace. Consider catching parsing errors and rethrowing 
an IllegalArgumentException with a clear message that includes the raw 
`spark.task.cpus` value and expected format (integer or decimal > 0).



##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
     val executorCores = SparkResourceUtil.getExecutorCores(conf)
     // spark.task.cpus is read raw here, which bypasses Spark's own 
checkValue(_ > 0) (and on
     // Spark < 4.2 that positivity check does not exist at all). getTaskSlots 
runs during driver
-    // plugin init, before Spark validates the value, so fail fast on a 
non-positive setting rather
-    // than dividing by it: a zero would throw an opaque "/ by zero" 
ArithmeticException and a
-    // negative would silently produce negative task slots and off-heap 
budgets.
-    val taskCores = conf.getInt("spark.task.cpus", 1)
-    require(taskCores > 0, s"spark.task.cpus should be positive, but was 
$taskCores")
+    // plugin init, before Spark validates the value. Spark 4.3 allows 
fractional task cpus
+    // (SPARK-58192), so read it as a decimal.
+    getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus", 
"1").trim))
+  }
+
+  /** Returns floor(executorCores / taskCpus), at least one, like Spark's own 
slot computation. */
+  def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {

Review Comment:
   `BigDecimal(conf.get(...))` will throw a NumberFormatException for malformed 
values (e.g., empty string, non-numeric), which will surface during plugin init 
with a low-signal stack trace. Consider catching parsing errors and rethrowing 
an IllegalArgumentException with a clear message that includes the raw 
`spark.task.cpus` value and expected format (integer or decimal > 0).



##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,18 @@ object Validators {
     }
   }
 
+  private object FallbackByUnsupportedSparkFeatures extends Validator {
+    override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+      // The scan reports `SinglePartition` and Spark may have elided a 
shuffle above it, while the
+      // native scan doesn't coalesce its splits into a single task.
+      case p: FileSourceScanExec if 
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>
+        fail(s"${p.nodeName} uses single-task execution, which is not 
supported")
+      case p: LocalTableScanExec if p.outputPartitioning.numPartitions == 1 =>
+        fail(s"${p.nodeName} uses single-task execution, which is not 
supported")
+      case _ => pass()
+    }
+  }

Review Comment:
   This introduces a new validation rule that forces fallback for single-task 
scans. Add unit tests that assert (1) validation fails for a 
`FileSourceScanExec` flagged by the shim and for single-partition 
`LocalTableScanExec`, and (2) validation passes for the normal 
(multi-partition) cases, to prevent regressions as Spark scan partitioning 
behavior evolves.



##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
     val executorCores = SparkResourceUtil.getExecutorCores(conf)
     // spark.task.cpus is read raw here, which bypasses Spark's own 
checkValue(_ > 0) (and on
     // Spark < 4.2 that positivity check does not exist at all). getTaskSlots 
runs during driver
-    // plugin init, before Spark validates the value, so fail fast on a 
non-positive setting rather
-    // than dividing by it: a zero would throw an opaque "/ by zero" 
ArithmeticException and a
-    // negative would silently produce negative task slots and off-heap 
budgets.
-    val taskCores = conf.getInt("spark.task.cpus", 1)
-    require(taskCores > 0, s"spark.task.cpus should be positive, but was 
$taskCores")
+    // plugin init, before Spark validates the value. Spark 4.3 allows 
fractional task cpus
+    // (SPARK-58192), so read it as a decimal.
+    getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus", 
"1").trim))
+  }
+
+  /** Returns floor(executorCores / taskCpus), at least one, like Spark's own 
slot computation. */
+  def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {
+    // Fail fast on a non-positive setting rather than dividing by it: a zero 
would throw an opaque
+    // ArithmeticException and a negative would silently produce negative task 
slots and off-heap
+    // budgets.
+    require(taskCpus > 0, s"spark.task.cpus should be positive, but was 
$taskCpus")

Review Comment:
   `BigDecimal(conf.get(...))` will throw a NumberFormatException for malformed 
values (e.g., empty string, non-numeric), which will surface during plugin init 
with a low-signal stack trace. Consider catching parsing errors and rethrowing 
an IllegalArgumentException with a clear message that includes the raw 
`spark.task.cpus` value and expected format (integer or decimal > 0).



##########
pom.xml:
##########
@@ -1549,6 +1549,101 @@
         </plugins>
       </build>
     </profile>
+    <profile>
+      <id>spark-4.3</id>
+      <properties>
+        <sparkbundle.version>4.3</sparkbundle.version>
+        
<sparkshim.artifactId>spark-sql-columnar-shims-spark43</sparkshim.artifactId>
+        <spark.version>4.3.0</spark.version>
+        <!-- Spark 4.3.0 uses Scala 2.13.18; override the shared scala-2.13
+             profile (2.13.17) for Spark 4.3 builds only. -->
+        <scala.version>2.13.18</scala.version>
+        <delta.package.name>delta-spark_4.2</delta.package.name>
+        <delta.version>${delta4.version}</delta.version>
+        <delta.binary.version>4</delta.binary.version>
+        <!-- TODO: No Spark 4.3 artifacts published for 
Delta/Iceberg/Hudi/Paimon yet. These
+             values are placeholders inherited from spark-4.2; the 
delta/iceberg/hudi/paimon
+             profiles are opt-in (activeByDefault=false) and are NOT enabled 
on the Spark 4.3
+             CI lane. -->

Review Comment:
   Same issue as in `tools/gluten-it/pom.xml`: the Spark 4.3 profile sets 
`delta.package.name` to `delta-spark_4.2`. Even if the profile isn't enabled in 
CI, this becomes a footgun for developers. Consider removing Delta-related 
properties from the Spark 4.3 profile entirely until real Spark 4.3 artifacts 
exist, or add an explicit build-time guard so enabling Delta on Spark 4.3 
produces a clear, actionable error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to