zhouyejoe commented on a change in pull request #34018:
URL: https://github.com/apache/spark/pull/34018#discussion_r710781414
##########
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##########
@@ -355,6 +355,11 @@ object SparkEnv extends Logging {
None
}
+ // Set the application attemptId in the ExternalShuffleClient is
applicable.
+ // If there is no attemptId assigned, set the attemptId to -1.
+ externalShuffleClient.foreach(
+ shuffleClient =>
shuffleClient.setAppAttemptId(conf.getInt(config.APP_ATTEMPT_ID.key, -1)))
Review comment:
I don't think we need to change the CoarseGrainedExecutorBackend to set
the attemptId. The _conf.set(APP_ATTEMPT_ID, attemptId) in SparkContext happens
before the Executor starts.
Executor will fetch the Spark confs from Driver, and Driver will respond the
confs.
The [SparkProperties here is lazy
val](https://github.com/apache/spark/blob/917d7dad4dcdbeac5094899fa9b7fffc67376cec/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L132),
so it will include all the conf changes made in Driver.
Then in Executor, after it [gets the Spark
confs](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L451)
from Driver, it will[ create the SparkEnv after
that](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L471).
Then the SparkEnv will have the attemptId in the conf.
The test in our Yarn cluster runs fine, so Executor should have the
attemptId set, otherwise, the push blocks from Executor will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
