HeartSaVioR commented on a change in pull request #23260: [SPARK-26311][CORE]
New feature: apply custom log URL pattern for executor log URLs in SHS
URL: https://github.com/apache/spark/pull/23260#discussion_r250032593
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
##########
@@ -241,15 +248,23 @@ private[yarn] class ExecutorRunnable(
}
}
- // Add log urls
+ // Add log urls, as well as executor attributes
container.foreach { c =>
+ val containerId = ConverterUtils.toString(c.getId)
+ val address = c.getNodeHttpAddress
+
+ env("SPARK_EXECUTOR_ATTRIBUTE_HTTP_SCHEME") = httpScheme
Review comment:
Yeah I followed the way how current log URLs are propagated, and the code
was aged for years so I simply guessed we may haven't been able to find better
alternatives.
I agreed we may be also able to figure out these information in executor
side, because we are doing the same thing from driver in cluster mode which
code is already running in container. One thing I'm afraid for making change
is, current approach can be leveraged for all of resource managers (a general
solution), whereas we may need to have (slightly) different implementation of
`CoarseGrainedExecutorBackend` once we take the case for YARN.
If we are OK to try tackling on this, overriding `extractLogUrls` and
`extractAttributes` seems the simplest way to achieve it. We might change
`CoarseGrainedExecutorBackend` to become abstract class to make clear two
methods should be open to override, but it's OK if we simply extend
`CoarseGrainedExecutorBackend` and have YARN's own executor backend which only
overrides two above methods.
Would love to hear your thought about this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
