squito commented on a change in pull request #23951: [SPARK-27038][CORE][YARN]
Re-implement RackResolver to reduce resolving time
URL: https://github.com/apache/spark/pull/23951#discussion_r264297189
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
##########
@@ -37,4 +46,77 @@ private[yarn] class SparkRackResolver {
RackResolver.resolve(conf, hostName).getNetworkLocation()
}
+ /**
+ * Added in SPARK-27038.
+ * This should be changed to `RackResolver.resolve(conf, hostNames)`
+ * in hadoop releases with YARN-9332.
+ */
+ def resolve(conf: Configuration, hostNames: List[String]): List[Node] = {
+ SparkRackResolver.coreResolve(conf, hostNames)
+ }
+}
+
+/**
+ * Utility to resolve the rack for hosts in an efficient manner.
+ * It will cache the rack for individual hosts to avoid
+ * repeatedly performing the same expensive lookup.
+ *
+ * Its logic refers [[org.apache.hadoop.yarn.util.RackResolver]] and enhanced.
+ * This will be unnecessary in hadoop releases with YARN-9332.
+ * With that, we could just directly use
[[org.apache.hadoop.yarn.util.RackResolver]].
+ * In the meantime, this is a re-implementation for spark's use.
+ */
+object SparkRackResolver extends Logging {
+ private var dnsToSwitchMapping: DNSToSwitchMapping = _
+ private var initCalled = false
+
+ private def init(conf: Configuration): Unit = {
+ if (!initCalled) {
+ initCalled = true
+ val dnsToSwitchMappingClass =
+
conf.getClass(CommonConfigurationKeysPublic.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY,
+ classOf[ScriptBasedMapping], classOf[DNSToSwitchMapping])
+ if
(classOf[ScriptBasedMapping].isAssignableFrom(dnsToSwitchMappingClass)) {
+ val numArgs =
conf.getInt(CommonConfigurationKeysPublic.NET_TOPOLOGY_SCRIPT_NUMBER_ARGS_KEY,
+
CommonConfigurationKeysPublic.NET_TOPOLOGY_SCRIPT_NUMBER_ARGS_DEFAULT)
+ logInfo(s"Setting spark.hadoop.net.topology.script.number.args with a
higher value " +
Review comment:
I dont' think there is any question that the rest of the patch is useful. I
think both marcelo and I are worried that this isn't really giving the user any
actionable info. We're just going to get questions about what to set it to.
10K? 1M? 1B? They'll see this msg no matter what they set it to, and they
still won't have any idea what else to set it to. Clearly, that hadoop code is
intentionally avoiding too large an arg list.
Note that if you used the de-duping I suggest, it would already help in your
use case. You have 19K tasks on 1K hosts. With 100 script args, and no
de-duping, that's 190 invocations of the script. With your patch &
spark.hadoop.net.topology.script.number.args=10K, its down to 2 script
invocations. With your patch + de-duping, and leaving
spark.hadoop.net.topology.script.number.args at the default of 100, you'd only
have 10 invocations. (perhaps each invocation of the script would be faster
when its got 100 args vs. 10K args?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
