Copilot commented on code in PR #13519:
URL: https://github.com/apache/gluten/pull/13519#discussion_r4505826284
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
}
}
+ private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+ override def computeValue(clazz: Class[_]): Method =
clazz.getMethod("shuffleBlockResolver")
Review Comment:
Using `Class#getMethod` only finds public methods. If Spark 4.3 exposes
`shuffleBlockResolver` on `BlockingShuffleManager` with non-public visibility
(or it’s inherited but not public), this will throw at runtime
(NoSuchMethodException / IllegalAccessException). To make this robust across
Spark builds, prefer `getDeclaredMethod` with `setAccessible(true)` (and/or try
`getMethod` then fall back to `getDeclaredMethod`) and wrap failures with a
clear error message that includes the shuffle manager class name.
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
}
}
+ private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+ override def computeValue(clazz: Class[_]): Method =
clazz.getMethod("shuffleBlockResolver")
+ }
+
+ /**
+ * Returns the block resolver of a registered shuffle manager. Spark 4.3
declares
+ * `shuffleBlockResolver` only on `BlockingShuffleManager` (which doesn't
exist before 4.3), so it
+ * is looked up reflectively to compile against every supported Spark
version.
+ */
+ private def shuffleBlockResolverOf(manager: ShuffleManager):
ShuffleBlockResolver = {
+ shuffleBlockResolverAccessor
+ .get(manager.getClass)
+ .invoke(manager)
+ .asInstanceOf[ShuffleBlockResolver]
+ }
Review Comment:
Using `Class#getMethod` only finds public methods. If Spark 4.3 exposes
`shuffleBlockResolver` on `BlockingShuffleManager` with non-public visibility
(or it’s inherited but not public), this will throw at runtime
(NoSuchMethodException / IllegalAccessException). To make this robust across
Spark builds, prefer `getDeclaredMethod` with `setAccessible(true)` (and/or try
`getMethod` then fall back to `getDeclaredMethod`) and wrap failures with a
clear error message that includes the shuffle manager class name.
##########
gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/GlutenCostEvaluator.scala:
##########
@@ -33,8 +33,20 @@ case class GlutenCostEvaluator() extends CostEvaluator with
SQLConfHelper {
val forceOptimizeSkewedJoin =
conf.getConfString("spark.sql.adaptive.forceOptimizeSkewedJoin").toBoolean
val clazz =
Utils.classForName("org.apache.spark.sql.execution.adaptive.SimpleCostEvaluator")
- val ctor = clazz.getConstructor(classOf[Boolean])
-
ctor.newInstance(forceOptimizeSkewedJoin.asInstanceOf[Object]).asInstanceOf[CostEvaluator]
+ if (SparkVersionUtil.gteSpark43) {
+ // Spark 4.3 added the `countLocalSort` parameter.
+ val countLocalSort =
+
conf.getConfString("spark.sql.adaptive.costEvaluator.countLocalSort.enabled").toBoolean
Review Comment:
`getConfString(key)` will throw if the key is absent, which can break
initialization across Spark patch versions or custom builds where this config
is missing/renamed. Prefer using a defaulted lookup (where available) or
guarding with `SparkConf.contains`/`SQLConf.contains` and defaulting to `false`
to keep the evaluator resilient.
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
val executorCores = SparkResourceUtil.getExecutorCores(conf)
// spark.task.cpus is read raw here, which bypasses Spark's own
checkValue(_ > 0) (and on
// Spark < 4.2 that positivity check does not exist at all). getTaskSlots
runs during driver
- // plugin init, before Spark validates the value, so fail fast on a
non-positive setting rather
- // than dividing by it: a zero would throw an opaque "/ by zero"
ArithmeticException and a
- // negative would silently produce negative task slots and off-heap
budgets.
- val taskCores = conf.getInt("spark.task.cpus", 1)
- require(taskCores > 0, s"spark.task.cpus should be positive, but was
$taskCores")
+ // plugin init, before Spark validates the value. Spark 4.3 allows
fractional task cpus
+ // (SPARK-58192), so read it as a decimal.
+ getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus",
"1").trim))
Review Comment:
`BigDecimal(conf.get(...))` will throw a NumberFormatException for malformed
values (e.g., empty string, non-numeric), which will surface during plugin init
with a low-signal stack trace. Consider catching parsing errors and rethrowing
an IllegalArgumentException with a clear message that includes the raw
`spark.task.cpus` value and expected format (integer or decimal > 0).
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
val executorCores = SparkResourceUtil.getExecutorCores(conf)
// spark.task.cpus is read raw here, which bypasses Spark's own
checkValue(_ > 0) (and on
// Spark < 4.2 that positivity check does not exist at all). getTaskSlots
runs during driver
- // plugin init, before Spark validates the value, so fail fast on a
non-positive setting rather
- // than dividing by it: a zero would throw an opaque "/ by zero"
ArithmeticException and a
- // negative would silently produce negative task slots and off-heap
budgets.
- val taskCores = conf.getInt("spark.task.cpus", 1)
- require(taskCores > 0, s"spark.task.cpus should be positive, but was
$taskCores")
+ // plugin init, before Spark validates the value. Spark 4.3 allows
fractional task cpus
+ // (SPARK-58192), so read it as a decimal.
+ getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus",
"1").trim))
+ }
+
+ /** Returns floor(executorCores / taskCpus), at least one, like Spark's own
slot computation. */
+ def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {
Review Comment:
`BigDecimal(conf.get(...))` will throw a NumberFormatException for malformed
values (e.g., empty string, non-numeric), which will surface during plugin init
with a low-signal stack trace. Consider catching parsing errors and rethrowing
an IllegalArgumentException with a clear message that includes the raw
`spark.task.cpus` value and expected format (integer or decimal > 0).
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,18 @@ object Validators {
}
}
+ private object FallbackByUnsupportedSparkFeatures extends Validator {
+ override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+ // The scan reports `SinglePartition` and Spark may have elided a
shuffle above it, while the
+ // native scan doesn't coalesce its splits into a single task.
+ case p: FileSourceScanExec if
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>
+ fail(s"${p.nodeName} uses single-task execution, which is not
supported")
+ case p: LocalTableScanExec if p.outputPartitioning.numPartitions == 1 =>
+ fail(s"${p.nodeName} uses single-task execution, which is not
supported")
+ case _ => pass()
+ }
+ }
Review Comment:
This introduces a new validation rule that forces fallback for single-task
scans. Add unit tests that assert (1) validation fails for a
`FileSourceScanExec` flagged by the shim and for single-partition
`LocalTableScanExec`, and (2) validation passes for the normal
(multi-partition) cases, to prevent regressions as Spark scan partitioning
behavior evolves.
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
val executorCores = SparkResourceUtil.getExecutorCores(conf)
// spark.task.cpus is read raw here, which bypasses Spark's own
checkValue(_ > 0) (and on
// Spark < 4.2 that positivity check does not exist at all). getTaskSlots
runs during driver
- // plugin init, before Spark validates the value, so fail fast on a
non-positive setting rather
- // than dividing by it: a zero would throw an opaque "/ by zero"
ArithmeticException and a
- // negative would silently produce negative task slots and off-heap
budgets.
- val taskCores = conf.getInt("spark.task.cpus", 1)
- require(taskCores > 0, s"spark.task.cpus should be positive, but was
$taskCores")
+ // plugin init, before Spark validates the value. Spark 4.3 allows
fractional task cpus
+ // (SPARK-58192), so read it as a decimal.
+ getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus",
"1").trim))
+ }
+
+ /** Returns floor(executorCores / taskCpus), at least one, like Spark's own
slot computation. */
+ def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {
+ // Fail fast on a non-positive setting rather than dividing by it: a zero
would throw an opaque
+ // ArithmeticException and a negative would silently produce negative task
slots and off-heap
+ // budgets.
+ require(taskCpus > 0, s"spark.task.cpus should be positive, but was
$taskCpus")
Review Comment:
`BigDecimal(conf.get(...))` will throw a NumberFormatException for malformed
values (e.g., empty string, non-numeric), which will surface during plugin init
with a low-signal stack trace. Consider catching parsing errors and rethrowing
an IllegalArgumentException with a clear message that includes the raw
`spark.task.cpus` value and expected format (integer or decimal > 0).
##########
pom.xml:
##########
@@ -1549,6 +1549,101 @@
</plugins>
</build>
</profile>
+ <profile>
+ <id>spark-4.3</id>
+ <properties>
+ <sparkbundle.version>4.3</sparkbundle.version>
+
<sparkshim.artifactId>spark-sql-columnar-shims-spark43</sparkshim.artifactId>
+ <spark.version>4.3.0</spark.version>
+ <!-- Spark 4.3.0 uses Scala 2.13.18; override the shared scala-2.13
+ profile (2.13.17) for Spark 4.3 builds only. -->
+ <scala.version>2.13.18</scala.version>
+ <delta.package.name>delta-spark_4.2</delta.package.name>
+ <delta.version>${delta4.version}</delta.version>
+ <delta.binary.version>4</delta.binary.version>
+ <!-- TODO: No Spark 4.3 artifacts published for
Delta/Iceberg/Hudi/Paimon yet. These
+ values are placeholders inherited from spark-4.2; the
delta/iceberg/hudi/paimon
+ profiles are opt-in (activeByDefault=false) and are NOT enabled
on the Spark 4.3
+ CI lane. -->
Review Comment:
Same issue as in `tools/gluten-it/pom.xml`: the Spark 4.3 profile sets
`delta.package.name` to `delta-spark_4.2`. Even if the profile isn't enabled in
CI, this becomes a footgun for developers. Consider removing Delta-related
properties from the Spark 4.3 profile entirely until real Spark 4.3 artifacts
exist, or add an explicit build-time guard so enabling Delta on Spark 4.3
produces a clear, actionable error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]