jerrypeng commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1566272683
##########
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##########
@@ -669,6 +753,27 @@ private[spark] class ExecutorAllocationManager(
private val stageAttemptToExecutorPlacementHints =
new mutable.HashMap[StageAttempt, (Int, Map[String, Int], Int)]
+ // to track total no. of tasks in each stage of a micro-batch (streaming
use case)
+ // this will help in requesting resources by counting pending tasks in job,
+ // rather than counting pending tasks in a stage.
+ private val stageIdToNumTasks = new mutable.HashMap[Int, Int]
Review Comment:
Are there any race conditions for updating this map? I am not an expert on
the DAGScheduler or ExecutorAllocationManager, but it seems to be more then one
thread will be reading/updating this map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
