pan3793 commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1960063444
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -1049,6 +1049,10 @@ private[spark] class Client(
javaOpts += s"-Djava.net.preferIPv6Addresses=${Utils.preferIPv6}"
+ sparkConf.getOption("spark.ml.allowNativeBlas").foreach { allowNativeBlas
=>
+ javaOpts += s"-Dspark.ml.allowNativeBlas=$allowNativeBlas"
Review Comment:
> Do other resource managers like k8s need this? not sure
K8s does not need that change.
#### Spark on YARN
the code appends `-Dspark.ml.allowNativeBlas=...` to YARN AM process
command, we should assemble a java command to let YARN RM know how to bootstrap
the process
#### Spark on K8s
- client mode, no driver Pod
- cluster mode, run `spark-submit`(which carries all Java options from local
`spark-submit`) in the driver Pod
so it does not need to append those Java options again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
