Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3121#issuecomment-62081957
Hey @andrewor14,
I've addressed your earlier review comments; please take another look.
Regarding automatic cleanup of failed contexts: my concern was that if if
thread attempting to create a new context notices that there's another context
being constructed by some other thread (e.g. `activeSparkContextCreationSite !=
None` and `activeContextIsFullyConstructed == false`, then it can't determine
whether the other thread was still in the body of the constructor (and running
slowly) or whether it has exited with an exception (since we don't catch the
exception). For a distributed systems analogy, this is like a case where we
haven't received a message and are unable to determine whether it has been lost
or is merely delayed.
To address the most common user-facing cases (e.g. interactive REPLs), I
suppose we could also record the id of the thread that is currently in the body
of the SparkContext constructor. If a creation attempt fails, then we can
safely initiate a re-creation attempt from the _same_ thread since we don't
need to worry about asynchrony.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
