Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22034#discussion_r208786793
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -169,6 +171,19 @@ private[spark] class Executor(
startDriverHeartbeater()
+ /**
+ * We add an empty WebUI in executor to enable Executor MetricsServlet
sink if needed.
+ */
+ private val executorWebUiEnabled =
conf.getBoolean("spark.executor.ui.enabled", false)
+ private[executor] var webUi: ExecutorWebUI = _
+ if (executorWebUiEnabled && !isLocal) {
+ webUi = new ExecutorWebUI(conf, env.securityManager,
SparkUI.getUIPort(conf))
+ webUi.bind()
+ env.metricsSystem.getServletHandlers.foreach(webUi.attachHandler)
+ heartbeatReceiverRef.ask[Boolean](ReportExecutorWebUrl(executorId,
webUi.webUrl))
+ logInfo(s"Starting executor web ui at ${webUi.webUrl}")
--- End diff --
Maybe you can use jmx or some other tools. Basically the metrics is still
report in every N seconds, so pulling frequently doesn't help increasing the
accuracy.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
