Many many thanks Andy
From: Nicholas Chammas <[email protected]> Date: Friday, September 26, 2014 at 11:24 AM To: Andrew Davidson <[email protected]> Cc: Davies Liu <[email protected]>, "[email protected]" <[email protected]> Subject: Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found > Are you able to use the regular PySpark shell on your EC2 cluster? That would > be the first thing to confirm is working. > > I don’t know whether the version of Python on the cluster would affect whether > IPython works or not, but if you want to try manually upgrading Python on a > cluster launched by spark-ec2, there are some instructions in the comments > here <https://issues.apache.org/jira/browse/SPARK-922> for doing so. > > Nick > > > > On Fri, Sep 26, 2014 at 2:18 PM, Andy Davidson <[email protected]> > wrote: >> Hi Davies >> >> The real issue is about cluster management. I am new to the spark world and >> am not a system administrator. It seem like the problem is with the >> spark-ec2 launch script. It is installing old version of python >> >> In the mean time I am trying to figure out how I can manually install the >> correct version on all the machines in my cluster >> >> Thanks >> >> Andy >> >> From: Davies Liu <[email protected]> >> Date: Thursday, September 25, 2014 at 9:58 PM >> To: Andrew Davidson <[email protected]> >> Cc: "[email protected]" <[email protected]> >> Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found >> >>> Maybe you have Python 2.7 on master but Python 2.6 in cluster, >>> you should upgrade python to 2.7 in cluster, or use python 2.6 in >>> master by set PYSPARK_PYTHON=python2.6 >>> >>> On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson >>> <[email protected]> wrote: >>>> Hi >>>> >>>> I am running into trouble using iPython notebook on my cluster. Use the >>>> following command to set the cluster up >>>> >>>> $ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE >>>> --region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME >>>> >>>> >>>> On master I launch python as follows >>>> >>>> $ IPYTHON_OPTS="notebook --pylab inline --no-browser --port=7000" >>>> $SPARK_HOME/bin/pyspark >>>> >>>> >>>> It looks like the problem is the cluster is using an old version of python >>>> and python. Any idea how I can easily upgrade ? The following version >>>> works >>>> on my mac >>>> >>>> Thanks >>>> >>>> Andy >>>> >>>> {'commit_hash': '681fd77', >>>> 'commit_source': 'installation', >>>> 'default_encoding': 'UTF-8', >>>> 'ipython_path': '/Library/Python/2.7/site-packages/IPython', >>>> 'ipython_version': '2.1.0', >>>> 'os_name': 'posix', >>>> 'platform': 'Darwin-13.3.0-x86_64-i386-64bit', >>>> 'sys_executable': '/usr/bin/python', >>>> 'sys_platform': 'darwin', >>>> 'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1 >>>> Compatible Apple LLVM 5.0 (clang-500.0.68)]’} >>>> >>>> >>>> >>>> >>> >
