yaooqinn opened a new pull request #25964: [SPARK-29287][Core] Add ExecutorConstructed message to tell driver which executor is ready for making offers URL: https://github.com/apache/spark/pull/25964 ### What changes were proposed in this pull request? Add ExecutorConstructed message and send it to the driver when the executor if fully constructed, then the driver can set the associated executor be ready for making offers. ### Why are the changes needed? The executors send RegisterExecutor messages to the driver when onStart. The driver put the executor data in “the ready to serve map” if it could be, then send RegisteredExecutor back to the executor. The driver now can make an offer to this executor. But the executor is not fully constructed yet. When it received RegisteredExecutor, it start to construct itself, initializing block manager, maybe register to the local shuffle server in the way of retrying, then start the heart beating to driver ... The task allocated here may fail if the executor fails to start or cannot get heart beating to the driver in time. Sometimes, even worse, when dynamic allocation and blacklisting is enabled and when the runtime executor number down to min executor setting, and those executors receive tasks before fully constructed and if any error happens, the application may be blocked or tear down. ### Does this PR introduce any user-facing change? NO ### How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. -->
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
