pgandhi999 commented on issue #24035: [SPARK-27112] : Spark Scheduler encounters two independent Deadlocks … URL: https://github.com/apache/spark/pull/24035#issuecomment-472032142 @squito Thank you for your response. I can explain more about the `makeOffersLock` here. PR https://github.com/apache/spark/pull/17091 introduced the part about acquiring synchronization on `CoarseGrainedSchedulerBackend` object in the method `makeOffers()`. This particular piece of code introduced a deadlock between task-result-getter thread and dispatcher-event-loop thread. I can simply removed the synchronized statement in `makeOffers()` and the deadlock would be resolved and we really do not need `makeOffersLock`. However, removing the synchronized statement will once again expose the race condition described in JIRA https://issues.apache.org/jira/browse/SPARK-19757 for which the fix in the corresponding PR was merged. `makeOffersLock` here serves as the solution to the above problem. By synchronizing on `makeOffersLock`, the race condition between dynamic-executor-allocation thread and dispatcher-event-loop thread is avoided. That is indeed it's sole purpose. I am however, open to discussing and working on better solutions to the above problem, if any. Thank you.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
