Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2624#issuecomment-57840767
Okay, this is a pretty significant change to remove the threadlocal object
completely. There are two things we can do
- Either we can do something minimal to just clear the reference, so that
repeated sparkContext creation works from pySpark.
- Or we do this scary refactoring and eliminate the threadlocal completely.
I know that there are probably no usecases where the threadlocal object on any
thread would be different from that of the global lastSparkEnv, but are we
entirely sure?
I think we need more input from others @pwendell @mateiz @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
