Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2379#discussion_r17696773
--- Diff:
core/src/main/scala/org/apache/spark/deploy/worker/WorkerArguments.scala ---
@@ -47,14 +48,15 @@ private[spark] class WorkerArguments(args:
Array[String], conf: SparkConf) {
if (System.getenv("SPARK_WORKER_WEBUI_PORT") != null) {
webUiPort = System.getenv("SPARK_WORKER_WEBUI_PORT").toInt
}
- if (conf.contains("spark.worker.ui.port")) {
- webUiPort = conf.get("spark.worker.ui.port").toInt
+ if (System.getenv("spark.worker.ui.port")!= null) {
+ webUiPort = System.getenv("spark.worker.ui.port").toInt
--- End diff --
Ha, sorry about that.
=== TRANSLATION ===
@witgo: In Spark, we do the following a lot: `val conf = new SparkConf`.
For example, SparkHadoopUtil.scala#L38 ,EventLoggingListener.scala#L216. This
requires us to set `system property`. This way, if we don't change the
initialization order, we need to set both `conf` and `sys.props`.
@andrewor14 Yes, I understand that part. What I mean is if we call
`conf.setIfMissing` after initializing `SparkConf`, then we don't need to set
the corresponding configs through `sys.props` *before* we initialize it. This
way we can preserve the existing initialization order. At the same time,
because `spark-defaults.conf` can also set `spark.worker.ui.port`, we need to
first load `spark-defaults.conf` before accessing
`conf.get("spark.worker.ui.port")`. All of this logic can be implemented in
`WorkerArgument`, then `Worker` doesn't need to change much. Do you understand
what I mean?
=== END ===
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
