Many many thanks

Andy

From:  Nicholas Chammas <[email protected]>
Date:  Friday, September 26, 2014 at 11:24 AM
To:  Andrew Davidson <[email protected]>
Cc:  Davies Liu <[email protected]>, "[email protected]"
<[email protected]>
Subject:  Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line
magic function `%matplotlib` not found

> Are you able to use the regular PySpark shell on your EC2 cluster? That would
> be the first thing to confirm is working.
> 
> I don’t know whether the version of Python on the cluster would affect whether
> IPython works or not, but if you want to try manually upgrading Python on a
> cluster launched by spark-ec2, there are some instructions in the comments
> here <https://issues.apache.org/jira/browse/SPARK-922>  for doing so.
> 
> Nick
> 
> ​
> 
> On Fri, Sep 26, 2014 at 2:18 PM, Andy Davidson <[email protected]>
> wrote:
>> Hi Davies
>> 
>> The real issue is about cluster management. I am new to the spark world and
>> am not a system administrator.  It seem like the problem is with the
>> spark-ec2 launch script. It is installing  old version of python
>> 
>> In the mean time I am trying to figure out how I can manually install the
>> correct version on all the machines in my cluster
>> 
>> Thanks
>> 
>> Andy
>> 
>> From:  Davies Liu <[email protected]>
>> Date:  Thursday, September 25, 2014 at 9:58 PM
>> To:  Andrew Davidson <[email protected]>
>> Cc:  "[email protected]" <[email protected]>
>> Subject:  Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
>> 
>>> Maybe you have Python 2.7 on master but Python 2.6 in cluster,
>>> you should upgrade python to 2.7 in cluster, or use python 2.6 in
>>> master by set PYSPARK_PYTHON=python2.6
>>> 
>>> On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
>>> <[email protected]> wrote:
>>>>  Hi
>>>> 
>>>>  I am running into trouble using iPython notebook on my cluster. Use the
>>>>  following command to set the cluster up
>>>> 
>>>>  $ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
>>>>  --region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
>>>> 
>>>> 
>>>>  On master I launch python as follows
>>>> 
>>>>  $ IPYTHON_OPTS="notebook --pylab inline --no-browser --port=7000"
>>>>  $SPARK_HOME/bin/pyspark
>>>> 
>>>> 
>>>>  It looks like the problem is the cluster is using an old version of python
>>>>  and python. Any idea how I can easily upgrade ? The following version
>>>> works
>>>>  on my mac
>>>> 
>>>>  Thanks
>>>> 
>>>>  Andy
>>>> 
>>>>  {'commit_hash': '681fd77',
>>>>   'commit_source': 'installation',
>>>>   'default_encoding': 'UTF-8',
>>>>   'ipython_path': '/Library/Python/2.7/site-packages/IPython',
>>>>   'ipython_version': '2.1.0',
>>>>   'os_name': 'posix',
>>>>   'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
>>>>   'sys_executable': '/usr/bin/python',
>>>>   'sys_platform': 'darwin',
>>>>   'sys_version': '2.7.5 (default, Mar  9 2014, 22:15:05) \n[GCC 4.2.1
>>>>  Compatible Apple LLVM 5.0 (clang-500.0.68)]’}
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
> 


Reply via email to