tgravescs commented on a change in pull request #28656:
URL: https://github.com/apache/spark/pull/28656#discussion_r431253949
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
##########
@@ -1107,10 +1107,16 @@ private[spark] class TaskSetManager(
def recomputeLocality(): Unit = {
// A zombie TaskSetManager may reach here while executorLost happens
if (isZombie) return
+ val previousLocalityIndex = currentLocalityIndex
val previousLocalityLevel = myLocalityLevels(currentLocalityIndex)
myLocalityLevels = computeValidLocalityLevels()
localityWaits = myLocalityLevels.map(getLocalityWait)
currentLocalityIndex = getLocalityIndex(previousLocalityLevel)
+ if (currentLocalityIndex > previousLocalityIndex) {
+ // SPARK-31837: there's new higher level locality, so shift to
+ // the highest locality level in terms of better data locality
+ currentLocalityIndex = 0
Review comment:
I have concerns with this. This is going to reset it back to highest
level all the time and then you will have to work your way back through the
levels, waiting at each level. This could really delay things a lot if you are
down to the ANY level and you have to wait 3 seconds between each one. The
problem you are describing is only on startup when you have no executors,
correct? Perhaps we can look at something more specific for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
