[
https://issues.apache.org/jira/browse/SPARK-1550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075509#comment-14075509
]
Josh Rosen commented on SPARK-1550:
-----------------------------------
Actually, there's still a similar problem in Spark 1.01:
{code}
>>> sc = SparkContext('sjkdlsjfsl')
14/07/26 16:04:14 INFO SecurityManager: Changing view acls to: joshrosen
14/07/26 16:04:14 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(joshrosen)
14/07/26 16:04:14 INFO Slf4jLogger: Slf4jLogger started
14/07/26 16:04:14 INFO Remoting: Starting remoting
14/07/26 16:04:14 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://spark@localhost:57189]
14/07/26 16:04:14 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://spark@localhost:57189]
14/07/26 16:04:14 INFO SparkEnv: Registering MapOutputTracker
14/07/26 16:04:14 INFO SparkEnv: Registering BlockManagerMaster
14/07/26 16:04:14 INFO DiskBlockManager: Created local directory at
/var/folders/d2/mt2f3tx14vgg3b38xq_3ydfw0000gp/T/spark-local-20140726160414-c9d9
14/07/26 16:04:14 INFO ConnectionManager: Bound socket to port 57190 with id =
ConnectionManagerId(localhost,57190)
14/07/26 16:04:14 INFO MemoryStore: MemoryStore started with capacity 297.0 MB
14/07/26 16:04:14 INFO BlockManagerMaster: Trying to register BlockManager
14/07/26 16:04:14 INFO BlockManagerMasterActor: Registering block manager
localhost:57190 with 297.0 MB RAM
14/07/26 16:04:14 INFO BlockManagerMaster: Registered BlockManager
14/07/26 16:04:14 INFO HttpFileServer: HTTP File server directory is
/var/folders/d2/mt2f3tx14vgg3b38xq_3ydfw0000gp/T/spark-a7453bad-5278-43fd-aa43-efb33228c660
14/07/26 16:04:14 INFO HttpServer: Starting HTTP Server
14/07/26 16:04:14 INFO SparkUI: Started SparkUI at http://localhost:4040
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/joshrosen/Documents/spark/spark/python/pyspark/context.py", line
133, in __init__
self._jsc = self._initialize_context(self._conf._jconf)
File "/Users/joshrosen/Documents/spark/spark/python/pyspark/context.py", line
179, in _initialize_context
return self._jvm.JavaSparkContext(jconf)
File
"/Users/joshrosen/Documents/spark/spark/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
line 669, in __call__
File
"/Users/joshrosen/Documents/spark/spark/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py",
line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling
None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.spark.SparkException: Could not parse Master URL: 'sjkdlsjfsl'
at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:1594)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:309)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:53)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:214)
at
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:722)
>>> sc = SparkContext('local')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/joshrosen/Documents/spark/spark/python/pyspark/context.py", line
93, in __init__
SparkContext._ensure_initialized(self, gateway=gateway)
File "/Users/joshrosen/Documents/spark/spark/python/pyspark/context.py", line
206, in _ensure_initialized
callsite.function, callsite.file, callsite.linenum))
ValueError: Cannot run multiple SparkContexts at once; existing
SparkContext(app=pyspark-shell, master=sjkdlsjfsl) created by __init__ at
<stdin>:1
>>>
{code}
The right solution is to clear {{active_spark_context}} if the SparkContext
creation fails. I'll submit a pull request for this in a little bit.
> Successive creation of spark context fails in pyspark, if the previous
> initialization of spark context had failed.
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-1550
> URL: https://issues.apache.org/jira/browse/SPARK-1550
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Reporter: Prabin Banka
> Labels: pyspark, sparkcontext
>
> For example;-
> In PySpark, if we try to initialize spark context with insufficient
> arguments, >>>sc=SparkContext('local')
> it fails with an exception
> Exception: An application name must be set in your configuration
> This is all fine.
> However, any successive creation of spark context with correct arguments,
> also fails,
> >>>s1=SparkContext('local', 'test1')
> AttributeError: 'SparkContext' object has no attribute 'master'
--
This message was sent by Atlassian JIRA
(v6.2#6252)
