akshaytayal commented on code in PR #13519:
URL: https://github.com/apache/gluten/pull/13519#discussion_r4507455109
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
}
}
+ private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+ override def computeValue(clazz: Class[_]): Method =
clazz.getMethod("shuffleBlockResolver")
Review Comment:
`shuffleBlockResolver` is a public member of Spark 4.3's
`BlockingShuffleManager` (`private[spark]` is public in bytecode), so
`getMethod` finds it on every implementing class, including inherited ones.
Posted by an AI assistant. Flag anything wrong here.
##########
gluten-core/src/main/scala/org/apache/spark/shuffle/ShuffleManagerRouter.scala:
##########
@@ -131,6 +135,22 @@ private object ShuffleManagerRouter {
}
}
+ private val shuffleBlockResolverAccessor = new ClassValue[Method] {
+ override def computeValue(clazz: Class[_]): Method =
clazz.getMethod("shuffleBlockResolver")
+ }
+
+ /**
+ * Returns the block resolver of a registered shuffle manager. Spark 4.3
declares
+ * `shuffleBlockResolver` only on `BlockingShuffleManager` (which doesn't
exist before 4.3), so it
+ * is looked up reflectively to compile against every supported Spark
version.
+ */
+ private def shuffleBlockResolverOf(manager: ShuffleManager):
ShuffleBlockResolver = {
+ shuffleBlockResolverAccessor
+ .get(manager.getClass)
+ .invoke(manager)
+ .asInstanceOf[ShuffleBlockResolver]
+ }
Review Comment:
`shuffleBlockResolver` is a public member of Spark 4.3's
`BlockingShuffleManager` (`private[spark]` is public in bytecode), so
`getMethod` finds it on every implementing class, including inherited ones.
Posted by an AI assistant. Flag anything wrong here.
##########
gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/GlutenCostEvaluator.scala:
##########
@@ -33,8 +33,20 @@ case class GlutenCostEvaluator() extends CostEvaluator with
SQLConfHelper {
val forceOptimizeSkewedJoin =
conf.getConfString("spark.sql.adaptive.forceOptimizeSkewedJoin").toBoolean
val clazz =
Utils.classForName("org.apache.spark.sql.execution.adaptive.SimpleCostEvaluator")
- val ctor = clazz.getConstructor(classOf[Boolean])
-
ctor.newInstance(forceOptimizeSkewedJoin.asInstanceOf[Object]).asInstanceOf[CostEvaluator]
+ if (SparkVersionUtil.gteSpark43) {
+ // Spark 4.3 added the `countLocalSort` parameter.
+ val countLocalSort =
+
conf.getConfString("spark.sql.adaptive.costEvaluator.countLocalSort.enabled").toBoolean
Review Comment:
`spark.sql.adaptive.costEvaluator.countLocalSort.enabled` is a registered
SQLConf entry in Spark 4.3, and this branch only runs on Spark 4.3+, so
`getConfString` returns its default when the key is unset.
Posted by an AI assistant. Flag anything wrong here.
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
val executorCores = SparkResourceUtil.getExecutorCores(conf)
// spark.task.cpus is read raw here, which bypasses Spark's own
checkValue(_ > 0) (and on
// Spark < 4.2 that positivity check does not exist at all). getTaskSlots
runs during driver
- // plugin init, before Spark validates the value, so fail fast on a
non-positive setting rather
- // than dividing by it: a zero would throw an opaque "/ by zero"
ArithmeticException and a
- // negative would silently produce negative task slots and off-heap
budgets.
- val taskCores = conf.getInt("spark.task.cpus", 1)
- require(taskCores > 0, s"spark.task.cpus should be positive, but was
$taskCores")
+ // plugin init, before Spark validates the value. Spark 4.3 allows
fractional task cpus
+ // (SPARK-58192), so read it as a decimal.
+ getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus",
"1").trim))
Review Comment:
A malformed `spark.task.cpus` is a fatal configuration error either way, and
the stack trace points at `SparkResourceUtil`, so no extra handling is added
here.
Posted by an AI assistant. Flag anything wrong here.
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
val executorCores = SparkResourceUtil.getExecutorCores(conf)
// spark.task.cpus is read raw here, which bypasses Spark's own
checkValue(_ > 0) (and on
// Spark < 4.2 that positivity check does not exist at all). getTaskSlots
runs during driver
- // plugin init, before Spark validates the value, so fail fast on a
non-positive setting rather
- // than dividing by it: a zero would throw an opaque "/ by zero"
ArithmeticException and a
- // negative would silently produce negative task slots and off-heap
budgets.
- val taskCores = conf.getInt("spark.task.cpus", 1)
- require(taskCores > 0, s"spark.task.cpus should be positive, but was
$taskCores")
+ // plugin init, before Spark validates the value. Spark 4.3 allows
fractional task cpus
+ // (SPARK-58192), so read it as a decimal.
+ getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus",
"1").trim))
+ }
+
+ /** Returns floor(executorCores / taskCpus), at least one, like Spark's own
slot computation. */
+ def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {
Review Comment:
A malformed `spark.task.cpus` is a fatal configuration error either way, and
the stack trace points at `SparkResourceUtil`, so no extra handling is added
here.
Posted by an AI assistant. Flag anything wrong here.
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala:
##########
@@ -79,15 +79,24 @@ object SparkResourceUtil extends Logging {
val executorCores = SparkResourceUtil.getExecutorCores(conf)
// spark.task.cpus is read raw here, which bypasses Spark's own
checkValue(_ > 0) (and on
// Spark < 4.2 that positivity check does not exist at all). getTaskSlots
runs during driver
- // plugin init, before Spark validates the value, so fail fast on a
non-positive setting rather
- // than dividing by it: a zero would throw an opaque "/ by zero"
ArithmeticException and a
- // negative would silently produce negative task slots and off-heap
budgets.
- val taskCores = conf.getInt("spark.task.cpus", 1)
- require(taskCores > 0, s"spark.task.cpus should be positive, but was
$taskCores")
+ // plugin init, before Spark validates the value. Spark 4.3 allows
fractional task cpus
+ // (SPARK-58192), so read it as a decimal.
+ getTaskSlots(executorCores, BigDecimal(conf.get("spark.task.cpus",
"1").trim))
+ }
+
+ /** Returns floor(executorCores / taskCpus), at least one, like Spark's own
slot computation. */
+ def getTaskSlots(executorCores: Int, taskCpus: BigDecimal): Int = {
+ // Fail fast on a non-positive setting rather than dividing by it: a zero
would throw an opaque
+ // ArithmeticException and a negative would silently produce negative task
slots and off-heap
+ // budgets.
+ require(taskCpus > 0, s"spark.task.cpus should be positive, but was
$taskCpus")
Review Comment:
A malformed `spark.task.cpus` is a fatal configuration error either way, and
the stack trace points at `SparkResourceUtil`, so no extra handling is added
here.
Posted by an AI assistant. Flag anything wrong here.
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/validator/Validators.scala:
##########
@@ -165,6 +170,18 @@ object Validators {
}
}
+ private object FallbackByUnsupportedSparkFeatures extends Validator {
+ override def validate(plan: SparkPlan): Validator.OutCome = plan match {
+ // The scan reports `SinglePartition` and Spark may have elided a
shuffle above it, while the
+ // native scan doesn't coalesce its splits into a single task.
+ case p: FileSourceScanExec if
SparkShimLoader.getSparkShims.isSingleTaskExecutionScan(p) =>
+ fail(s"${p.nodeName} uses single-task execution, which is not
supported")
+ case p: LocalTableScanExec if p.outputPartitioning.numPartitions == 1 =>
+ fail(s"${p.nodeName} uses single-task execution, which is not
supported")
+ case _ => pass()
+ }
+ }
Review Comment:
The Spark 4.3 `MarkSingleTaskExecutionSuite` runs under Gluten and executes
single-task file scans end to end. The `LocalTableScanExec` case applies only
when `spark.gluten.sql.columnar.localTableScan` is enabled (off by default).
Posted by an AI assistant. Flag anything wrong here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]