Makes sense. I'll give it a shot and check back here if that doesn't work. Are there plans to upgrade the EC2 deployment scripts and/or AMI to have Python 2.7 by default? If so, is there a ticket somewhere I can follow?
Nick On Thu, Feb 27, 2014 at 6:50 PM, Bryn Keller <[email protected]> wrote: > Hi Nick, > > All the nodes of the cluster need to have the same Python setup (path and > version). So if, e.g. you start running in 2.7.5 on the master and it ships > code to nodes that have 2.6.x, you'll get invalid opcode errors. > > Thanks, > Bryn > > > On Thu, Feb 27, 2014 at 3:48 PM, nicholas.chammas < > [email protected]> wrote: > >> The provided Spark EC2 >> scripts<https://spark.incubator.apache.org/docs/0.9.0/ec2-scripts.html>and >> default AMI ship with Python 2.6.8. >> >> I would like to use Python 2.7.5 or later. I believe that among the 2.x >> versions, 2.7 is the most popular. >> >> What's the easiest way to get my Spark cluster on Python 2.7 and working >> correctly? >> >> For starters, I naively upgraded Python on the master node using the >> instructions >> here<http://www.lecloud.net/post/61401763496/install-update-to-python-2-7-and-latest-pip-on-ec2>. >> The pyspark shell started up fine but a basic map on an RDD that used to >> work fine failed. Perhaps I just need to upgrade all the nodes in my >> cluster, and perhaps it's not that simple. I thought I'd check here first. >> >> Nick >> >> >> ------------------------------ >> View this message in context: Running Spark with Python >> 2.7.5+<http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-with-Python-2-7-5-tp2131.html> >> Sent from the Apache Spark User List mailing list >> archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com. >> > >
