wypoon commented on a change in pull request #23767: [SPARK-26329][CORE] Faster
polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r270529492
##########
File path: core/src/main/scala/org/apache/spark/executor/Executor.scala
##########
@@ -191,9 +191,20 @@ private[spark] class Executor(
*/
private val HEARTBEAT_INTERVAL_MS = conf.get(EXECUTOR_HEARTBEAT_INTERVAL)
+ /**
+ * Interval to poll for executor metrics, in milliseconds
+ */
+ private val METRICS_POLLING_INTERVAL_MS =
conf.get(EXECUTOR_METRICS_POLLING_INTERVAL)
Review comment:
3 is an independent thing. The problem before this change is that the only
way to get faster polling is to increase 3, which is abusing it. Now, the user
doesn't need to do anything to 3 at all (unless for some independent reason
they want to change the heartbeat interval). They tune 4 instead. So I have not
actually increased the number of configs the user needs to touch.
I do agree that the only cases I can think of where a user would want to set
4 are when they want to set 1 to true as well, but then the same can be said of
2. When a user sets 2 to true, they would want to set 1 to true as well.
Nevertheless, I don't think we want to couple the configs; I think it is still
necessary to have separate configs and let the user set them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
