This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 562977928772 [SPARK-49786][K8S] Lower
`KubernetesClusterSchedulerBackend.onDisconnected` log level to debug
562977928772 is described below
commit 5629779287724a891c81b16f982f9529bd379c39
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Sep 25 22:34:35 2024 -0700
[SPARK-49786][K8S] Lower `KubernetesClusterSchedulerBackend.onDisconnected`
log level to debug
### What changes were proposed in this pull request?
This PR aims to lower `KubernetesClusterSchedulerBackend.onDisconnected`
log level to debug.
### Why are the changes needed?
This INFO-level message was added here. We already propagate the
disconnection reason to UI, and `No executor found` has been used when an
unknown peer is connect or disconnect.
- https://github.com/apache/spark/pull/37821
The driver can be accessed by non-executors by design. And, all other
resource managers do not complain at INFO level.
```
INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: No
executor found for x.x.x.0:x
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review because this is a log level change.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48249 from dongjoon-hyun/SPARK-49786.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index 4e4634504a0f..09faa2a7fb1b 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -32,7 +32,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.submit.KubernetesClientUtils
import org.apache.spark.deploy.security.HadoopDelegationTokenManager
-import org.apache.spark.internal.LogKeys.{COUNT, HOST_PORT, TOTAL}
+import org.apache.spark.internal.LogKeys.{COUNT, TOTAL}
import org.apache.spark.internal.MDC
import
org.apache.spark.internal.config.SCHEDULER_MIN_REGISTERED_RESOURCES_RATIO
import org.apache.spark.resource.ResourceProfile
@@ -356,7 +356,7 @@ private[spark] class KubernetesClusterSchedulerBackend(
execIDRequester -= rpcAddress
// Expected, executors re-establish a connection with an ID
case _ =>
- logInfo(log"No executor found for ${MDC(HOST_PORT, rpcAddress)}")
+ logDebug(s"No executor found for ${rpcAddress}")
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
