Liang-Chi Hsieh created SPARK-19055:
---------------------------------------
Summary: SparkSession initialization will be associated with
invalid SparkContext when new SparkContext is created to replace stopped
SparkContext
Key: SPARK-19055
URL: https://issues.apache.org/jira/browse/SPARK-19055
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Reporter: Liang-Chi Hsieh
In SparkSession initialization, we store created the instance of SparkSession
into a class variable _instantiatedContext. Next time we can use
SparkSession.builder.getOrCreate() to retrieve the existing SparkSession
instance.
However, when the active SparkContext is stopped and we create another new
SparkContext to use, the existing SparkSession is still associated with the
stopped SparkContext. So the operations with this existing SparkSession will be
failed.
We need to detect such case in SparkSession and renew the class variable
_instantiatedContext if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
