This is an automated email from the ASF dual-hosted git repository.
mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new df7447b [SPARK-37712][YARN] Spark request yarn cluster metrics slow
cause delay
df7447b is described below
commit df7447bc62052e3d7391ba23d7220fb8c9b923fd
Author: Angerszhuuuu <[email protected]>
AuthorDate: Mon Jan 17 18:23:17 2022 -0600
[SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
### What changes were proposed in this pull request?
Spark will request yarn cluster metrics and print a log about nodemanager
number, it's not so important and this rpc is always slow
We can make it as debug level
### Why are the changes needed?
Avoid unnecessary delay when submit application.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Not need
Closes #34982 from AngersZhuuuu/SPARK-37712.
Authored-by: Angerszhuuuu <[email protected]>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
.../yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
index ca4fbbb..ae85ea8 100644
---
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
+++
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@@ -183,8 +183,10 @@ private[spark] class Client(
yarnClient.init(hadoopConf)
yarnClient.start()
- logInfo("Requesting a new application from cluster with %d NodeManagers"
- .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+ if (log.isDebugEnabled) {
+ logDebug("Requesting a new application from cluster with %d
NodeManagers"
+ .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+ }
// Get a new application from our RM
val newApp = yarnClient.createApplication()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
