Hi I am using Aanonda Python. Is there any way to specify the Python which we have o use for running pyspark in a cluster.
Best regards Jagan On Tue, Dec 30, 2014 at 6:27 PM, Eric Friedman <eric.d.fried...@gmail.com> wrote: > The Python installed in your cluster is 2.5. You need at least 2.6. > > ---- > Eric Friedman > > > On Dec 30, 2014, at 7:45 AM, Jaggu <jagana...@gmail.com> wrote: > > > > Hi Team, > > > > I was trying to execute a Pyspark code in cluster. It gives me the > following > > error. (Wne I run the same job in local it is working fine too :-() > > > > Eoor > > > > Error from python worker: > > /usr/lib/spark-1.2.0-bin-hadoop2.3/python/pyspark/context.py:209: > Warning: > > 'with' will become a reserved keyword in Python 2.6 > > Traceback (most recent call last): > > File > > > "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/runpy.py", > > line 85, in run_module > > loader = get_loader(mod_name) > > File > > > "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/pkgutil.py", > > line 456, in get_loader > > return find_loader(fullname) > > File > > > "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/pkgutil.py", > > line 466, in find_loader > > for importer in iter_importers(fullname): > > File > > > "/home/beehive/toolchain/x86_64-unknown-linux-gnu/python-2.5.2/lib/python2.5/pkgutil.py", > > line 422, in iter_importers > > __import__(pkg) > > File "/usr/lib/spark-1.2.0-bin-hadoop2.3/python/pyspark/__init__.py", > > line 41, in <module> > > from pyspark.context import SparkContext > > File "/usr/lib/spark-1.2.0-bin-hadoop2.3/python/pyspark/context.py", > > line 209 > > with SparkContext._lock: > > ^ > > SyntaxError: invalid syntax > > PYTHONPATH was: > > > > > /usr/lib/spark-1.2.0-bin-hadoop2.3/python:/usr/lib/spark-1.2.0-bin-hadoop2.3/python/lib/py4j-0.8.2.1-src.zip:/usr/lib/spark-1.2.0-bin-hadoop2.3/lib/spark-assembly-1.2.0-hadoop2.3.0.jar:/usr/lib/spark-1.2.0-bin-hadoop2.3/sbin/../python/lib/py4j-0.8.2.1-src.zip:/usr/lib/spark-1.2.0-bin-hadoop2.3/sbin/../python:/home/beehive/bin/utils/primitives:/home/beehive/bin/utils/pylogger:/home/beehive/bin/utils/asterScript:/home/beehive/bin/lib:/home/beehive/bin/utils/init:/home/beehive/installer/packages:/home/beehive/ncli > > java.io.EOFException > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > at > > > org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:163) > > at > > > org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:86) > > at > > > org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:62) > > at > org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:102) > > at > org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) > > at > > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > > at org.apache.spark.scheduler.Task.run(Task.scala:56) > > at > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:722) > > > > 14/12/31 04:49:58 INFO TaskSetManager: Starting task 0.1 in stage 0.0 > (TID > > 1, aster4, NODE_LOCAL, 1321 bytes) > > 14/12/31 04:49:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in > memory > > on aster4:43309 (size: 3.8 KB, free: 265.0 MB) > > 14/12/31 04:49:59 INFO TaskSetManager: Lost task 0.1 in stage 0.0 (TID > 1) on > > executor aster4: org.apache.spark.SparkException ( > > > > > > Any clue how to resolve the same. > > > > Best regards > > > > Jagan > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-with-error-from-PySpark-tp20907.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > -- ********************************** JAGANADH G http://jaganadhg.in *ILUGCBE* http://ilugcbe.org.in
