Pat McDonough created SPARK-1670:
------------------------------------
Summary: PySpark Fails to Create SparkContext Due To Debugging
Options in conf/java-opts
Key: SPARK-1670
URL: https://issues.apache.org/jira/browse/SPARK-1670
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.0.0
Environment: pats-air:spark pat$ IPYTHON=1 bin/pyspark
Python 2.7.5 (default, Aug 25 2013, 00:04:04)
...
IPython 1.1.0
...
Spark version 1.0.0-SNAPSHOT
Using Python version 2.7.5 (default, Aug 25 2013 00:04:04)
Reporter: Pat McDonough
When JVM debugging options are in conf/java-opts, it causes pyspark to fail
when creating the SparkContext. The java-opts file looks like the following:
{code}-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
{code}
Here's the error:
{code}---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/Library/Python/2.7/site-packages/IPython/utils/py3compat.pyc in
execfile(fname, *where)
202 else:
203 filename = fname
--> 204 __builtin__.execfile(filename, *where)
/Users/pat/Projects/spark/python/pyspark/shell.py in <module>()
41 SparkContext.setSystemProperty("spark.executor.uri",
os.environ["SPARK_EXECUTOR_URI"])
42
---> 43 sc = SparkContext(os.environ.get("MASTER", "local[*]"), "PySparkShell",
pyFiles=add_files)
44
45 print("""Welcome to
/Users/pat/Projects/spark/python/pyspark/context.pyc in __init__(self, master,
appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway)
92 tempNamedTuple = namedtuple("Callsite", "function file
linenum")
93 self._callsite = tempNamedTuple(function=None, file=None,
linenum=None)
---> 94 SparkContext._ensure_initialized(self, gateway=gateway)
95
96 self.environment = environment or {}
/Users/pat/Projects/spark/python/pyspark/context.pyc in
_ensure_initialized(cls, instance, gateway)
172 with SparkContext._lock:
173 if not SparkContext._gateway:
--> 174 SparkContext._gateway = gateway or launch_gateway()
175 SparkContext._jvm = SparkContext._gateway.jvm
176 SparkContext._writeToFile =
SparkContext._jvm.PythonRDD.writeToFile
/Users/pat/Projects/spark/python/pyspark/java_gateway.pyc in launch_gateway()
44 proc = Popen(command, stdout=PIPE, stdin=PIPE)
45 # Determine which ephemeral port the server started on:
---> 46 port = int(proc.stdout.readline())
47 # Create a thread to echo output from the GatewayServer, which is
required
48 # for Java log output to show up:
ValueError: invalid literal for int() with base 10: 'Listening for transport
dt_socket at address: 5005\n'
{code}
Note that when you use JVM debugging, the very first line of output (e.g. when
running spark-shell) looks like this:
{code}Listening for transport dt_socket at address: 5005{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
