LantaoJin commented on a change in pull request #23951:
[SPARK-13704][CORE][YARN] Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r270745197
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -811,8 +818,38 @@ private[spark] class TaskSchedulerImpl(
blacklistTrackerOpt.map(_.nodeBlacklist()).getOrElse(scala.collection.immutable.Set())
}
+ // Add a on-off switch to save time for rack resolving
+ private def skipRackResolving: Boolean = sc.conf.get(LOCALITY_WAIT_RACK) ==
0L
Review comment:
I knew that. Isolation deployment of computing and storage is more and more
popular in industry. If someone set `LOCALITY_WAIT` to 0 on purpose, almost
time it is no need data locality at all (especially rack level). Rack resolving
and locality algorithm still spends time on this isolation deployment case.
Could we open a new ticket to address this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
