holdenk commented on a change in pull request #29367:
URL: https://github.com/apache/spark/pull/29367#discussion_r468206734
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala
##########
@@ -298,6 +323,7 @@ private[spark] class ExecutorMonitor(
//
// This means that an executor may be marked as having shuffle data, and
thus prevented
// from being removed, even though the data may not be used.
+ // TODO: Only track used files (SPARK-31974)
Review comment:
Yes, since we're eventually going to want to use intelligent metrics to
decide who to scale down I'd like us to only track shuffle files that are being
used not speculative ones. Doesn't need to be addressed right now which is why
it's a TODO.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
