sunchao commented on code in PR #57525:
URL: https://github.com/apache/spark/pull/57525#discussion_r3659740040
##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -247,8 +247,12 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
override def receiveAndReply(context: RpcCallContext):
PartialFunction[Any, Unit] = {
case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls,
- attributes, resources, resourceProfileId) =>
- if (executorDataMap.contains(executorId)) {
+ attributes, resources, resourceProfileId, appId) =>
+ if (Option(appId).exists(_ != scheduler.sc.applicationId)) {
Review Comment:
[P1] Compare executor identity against the initialized scheduler application
ID
`SparkContext` sets `_applicationId` only after `_taskScheduler.start()`
returns, but this handler uses `scheduler.sc.applicationId` while that startup
is still in progress. In YARN client mode, `YarnClientSchedulerBackend.start()`
has already called `bindToYarn` with the correct application ID before entering
`waitForApplication`; `Client.monitorApplication` sleeps for
`spark.yarn.clientLaunchMonitorInterval` (one second by default) before its
first RM poll. During that window the application master initializes its
allocator and can launch legitimate executors carrying the correct non-null
YARN application ID. This check compares their valid ID with `null`, rejects
registration, and causes the executor to exit. YARN counts those exits against
its executor-failure limit and can fail the application, especially with a
longer launch monitor interval or `spark.executor.maxNumFailures=1`. Please
compare against the backend's already-bound `scheduler.applicationId()` (and
defer registr
ation if that authoritative ID is not available) instead of the late-published
`scheduler.sc.applicationId`. Add a regression that registers a legitimate
YARN-client executor before SparkContext publishes `_applicationId`; the
current tests all create a fully initialized SparkContext first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
