pan3793 commented on code in PR #57332:
URL: https://github.com/apache/spark/pull/57332#discussion_r3627777534
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -84,10 +100,19 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
val path = System.getenv("PATH") + File.pathSeparator +
SparkFiles.getRootDirectory()
builder.environment().put("PATH", path)
- // if OMP_NUM_THREADS is not explicitly set, override it with the value of
"spark.task.cpus"
- if (System.getenv("OMP_NUM_THREADS") == null) {
- builder.environment().put("OMP_NUM_THREADS",
conf.getConfString("spark.task.cpus", "1"))
- }
+ // Derive the script process's OMP_NUM_THREADS from the task's own cpu
amount, which honors
+ // stage-level resource profiles (a script transform can be a child of
mapInPandas/mapInArrow
+ // carrying a different profile), rounded up to an integer >= 1. We check
the config rather
+ // than System.getenv for an explicit user override, because Spark seeds
the executor's
+ // OMP_NUM_THREADS with the global default, so a getenv check would always
suppress the
+ // per-stage value.
+ val ompUserOverride =
+
Option(SparkEnv.get).exists(_.conf.contains("spark.executorEnv.OMP_NUM_THREADS"))
Review Comment:
After a closer look, this is another pre-existing issue, and `PythonRunner`
is also affected.
pre‑PR:
| | guard
| preserves inherited OMP?
|per‑stage correction? |
| - | -
| -
| - |
|PythonRunner
|conf.getOption("spark.executorEnv.OMP_NUM_THREADS").isEmpty |only
spark.executorEnv.* |no (used global)
|
|BaseScriptTransformationExec |System.getenv("OMP_NUM_THREADS") ==
null |any inherited value (pod template, image,
exported env) |no |
the key issue here is, `SparkContext` writes its own automatically generated
global default into the same `OMP_NUM_THREADS` env var, so we can not
distinguish it from the user-provided overrides
I'd like to restore `BaseScript`'s original
`System.getenv("OMP_NUM_THREADS") == null` guard, just made fractional-aware in
this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
