Hi Davies The real issue is about cluster management. I am new to the spark world and am not a system administrator. It seem like the problem is with the spark-ec2 launch script. It is installing old version of python
In the mean time I am trying to figure out how I can manually install the correct version on all the machines in my cluster Thanks Andy From: Davies Liu <dav...@databricks.com> Date: Thursday, September 25, 2014 at 9:58 PM To: Andrew Davidson <a...@santacruzintegration.com> Cc: "user@spark.apache.org" <user@spark.apache.org> Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found > Maybe you have Python 2.7 on master but Python 2.6 in cluster, > you should upgrade python to 2.7 in cluster, or use python 2.6 in > master by set PYSPARK_PYTHON=python2.6 > > On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson > <a...@santacruzintegration.com> wrote: >> Hi >> >> I am running into trouble using iPython notebook on my cluster. Use the >> following command to set the cluster up >> >> $ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE >> --region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME >> >> >> On master I launch python as follows >> >> $ IPYTHON_OPTS="notebook --pylab inline --no-browser --port=7000" >> $SPARK_HOME/bin/pyspark >> >> >> It looks like the problem is the cluster is using an old version of python >> and python. Any idea how I can easily upgrade ? The following version works >> on my mac >> >> Thanks >> >> Andy >> >> {'commit_hash': '681fd77', >> 'commit_source': 'installation', >> 'default_encoding': 'UTF-8', >> 'ipython_path': '/Library/Python/2.7/site-packages/IPython', >> 'ipython_version': '2.1.0', >> 'os_name': 'posix', >> 'platform': 'Darwin-13.3.0-x86_64-i386-64bit', >> 'sys_executable': '/usr/bin/python', >> 'sys_platform': 'darwin', >> 'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1 >> Compatible Apple LLVM 5.0 (clang-500.0.68)]ยน} >> >> >> >> >