vanzin commented on a change in pull request #23260: [SPARK-26311][CORE] New
feature: apply custom log URL pattern for executor log URLs in SHS
URL: https://github.com/apache/spark/pull/23260#discussion_r250016305
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
##########
@@ -241,15 +248,23 @@ private[yarn] class ExecutorRunnable(
}
}
- // Add log urls
+ // Add log urls, as well as executor attributes
container.foreach { c =>
+ val containerId = ConverterUtils.toString(c.getId)
+ val address = c.getNodeHttpAddress
+
+ env("SPARK_EXECUTOR_ATTRIBUTE_HTTP_SCHEME") = httpScheme
Review comment:
The way these attributes are being propagated is a bit weird, and limiting
in the long run. The driver needs to be able to figure them out, and the
executor is just piping the env variables back to the driver.
I don't really have a suggestion for a different way, but just wanted to
point out that this is a bit weird. e.g. in YARN the executor can figure out
all this info by itself; there just isn't a programmatic interface for it to
send that information to the driver as part of the registration message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
