[
https://issues.apache.org/jira/browse/SPARK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15444961#comment-15444961
]
Rahul Jain commented on SPARK-17261:
------------------------------------
i looked into 'zjffdu' pull request and the changed code. From my
understanding, this code change particularly addresses to SparkSession only and
doesn't provide backward compatibility for previous spark version codes(Spark
Context). I ran the below code in Spark 2.0 and it failed.
"""
>>> sc.stop()
>>> sc._instantiatedContext = None
>>> from pyspark import SparkContext
>>> from pyspark import HiveContext
>>> sc = SparkContext()
16/08/29 06:20:18 WARN Utils: Service 'SparkUI' could not bind on port 4040.
Attempting port 4041.
16/08/29 06:20:18 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
is set, falling back to uploading libraries under SPARK_HOME.
>>> sqlContext = HiveContext(sc)
>>> sqlContext.sql("show databases").collect()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/spark/python/pyspark/sql/context.py", line 350, in sql
return self.sparkSession.sql(sqlQuery)
File "/usr/lib/spark/python/pyspark/sql/session.py", line 541, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/usr/lib/spark/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
line 933, in __call__
File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/lib/spark/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py", line
312, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o44.sql.
: java.lang.IllegalStateException: Cannot call methods on a stopped
SparkContext.
This stopped SparkContext was created at:
"""
> Using HiveContext after re-creating SparkContext in Spark 2.0 throws
> "Java.lang.illegalStateException: Cannot call methods on a stopped
> sparkContext"
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17261
> URL: https://issues.apache.org/jira/browse/SPARK-17261
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.0.0
> Environment: Amazon AWS EMR 5.0
> Reporter: Rahul Jain
> Fix For: 2.0.0
>
>
> After stopping SparkSession if we recreate it and use HiveContext in it. it
> will throw error.
> Steps to reproduce:
> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
> spark.sql("show databases")
> spark.stop()
> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
> spark.sql("show databases")
> "Java.lang.illegalStateException: Cannot call methods on a stopped
> sparkContext"
> Above error occurs only in case of Pyspark not in SparkShell
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
