LuciferYang opened a new pull request, #12617: URL: https://github.com/apache/gluten/pull/12617
### What changes were proposed in this pull request? `SparkResourceUtil.getTaskSlots` computed `executorCores / taskCores` with no guard. `GlutenDriverPlugin.init` reads the slot count and divides by it, and four other callers (`MemoryTargets`, `ColumnarShuffleWriter`, and the Celeborn and Uniffle writers) use it as a denominator too. Two invalid configs crash there with an opaque error. When `spark.task.cpus > spark.executor.cores`, the integer division gives 0, so a caller's `offHeapSize / taskSlots` throws `ArithmeticException: / by zero`. When `spark.task.cpus = 0`, `getTaskSlots` itself throws, because it reads the value with raw `conf.getInt`, which skips Spark's `CPUS_PER_TASK.checkValue(_ > 0)` (that check runs only on the typed `conf.get(CPUS_PER_TASK)`). The plugin runs at the `PluginContainer` step in `SparkContext`, before `createTaskScheduler`. So Gluten throws before Spark's own validation (`validateTaskCpusLargeEnough` and the `CPUS_PER_TASK` check) can report the real problem, and the user gets a Gluten `ArithmeticException` stack trace instead of Spark's clear message. Both configs are invalid and Spark rejects them regardless, so the job does not start either way. The point is that Gluten should not misattribute the failure to itself with a divide-by-zero. `getTaskSlots` now returns a single slot when `taskCores <= 0`, and floors the quotient at 1 otherwise, so it never divides by a non-positive value and never returns 0. Gluten then defers to Spark for the error message. ### How was this patch tested? Added `SparkResourceUtilSuite` with four tests: `task.cpus` greater than executor cores floors to one slot, `task.cpus=0` does not divide by zero, `8 / 2` gives 4, and the default is one slot per core. The first two fail on the current code (they hit the two divide-by-zero paths) and pass after the fix. Closes #12616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
