GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/6006
[SPARK-7478][SQL] Added SQLContext.getOrCreate
Having a SQLContext singleton would make it easier for applications to use
a lazily instantiated single shared instance of SQLContext when needed. It
would avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770]. Also to get around this problem I had to suggest
creating a singleton instance -
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf.
@rxin @marmbrus
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-7478
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6006.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6006
----
commit bc72868f5e731b20eb2abeacec77203f019f21cd
Author: Tathagata Das <[email protected]>
Date: 2015-05-08T08:10:59Z
Added SQLContext.getOrCreate
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
