Hi Nicholas
I wrote some test code and found a way to get my matplotlib script to work
with the out of the box cluster created by spark-ec2
1. I commented out the python inline magic
#%matplotlib inline
2. Replace
#clear_output(wait=True)
clear_output(True)
The instructions of upgrading python you post on are very helpful.
https://issues.apache.org/jira/browse/SPARK-922
I got a response from the ipthon-dev mail list about the matplotlib upgrade
problem
“The issue is that we are requiring freetype 2.4 when we only need 2.3.
This has been fixed on both the maintenance branch and master and will
be included in 1.4.1 (which we hope to get out in the next week).
See https://github.com/matplotlib/matplotlib/issues/3413.
Tom”
Here is some back ground info about the versions installed out of the box
##
## python version on mac
##
import IPython
print IPython.sys_info()
'ipython_path': '/Library/Python/2.7/site-packages/IPython',
'ipython_version': '2.1.0',
sys_version': '2.7.5
import matplotlib
matplotlib.__version__
'1.1.1'
matplotlib.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/
matplotlib/__init__.pyc'
##
## ec2 cluster version before upgrade
##
import IPython
print IPython.sys_info()
'ipython_version': '0.13.2',
'sys_executable': '/usr/bin/python2.6',
'sys_version': '2.6.8 (unknown, Mar 14 2013, 09:31:22) \n[GCC 4.6.2
20111027 (Red Hat 4.6.2-2)]'}
import matplotlib
matplotlib.__version__
'0.99.1.1'
matplotlib.__file__
'/usr/lib64/python2.6/site-packages/matplotlib/__init__.pyc'
From: Nicholas Chammas <nicholas.cham...@gmail.com>
Date: Saturday, September 27, 2014 at 4:49 PM
To: Andrew Davidson <a...@santacruzintegration.com>
Cc: "user@spark.apache.org" <user@spark.apache.org>
Subject: Re: iPython notebook ec2 cluster matlabplot not found?
> Can you first confirm that the regular PySpark shell works on your cluster?
> Without upgrading to 2.7. That is, you log on to your master using spark-ec2
> login and run bin/pyspark successfully without any special flags.
>
> And as far as I can tell, you should be able to use IPython at 2.6, so I’d
> next confirm that that is working before throwing the 2.7 upgrade into the
> mix.
>
> Also, when upgrading or installing things, try doing so for all the nodes in
> your cluster using pssh. If you install stuff just on the master without
> somehow transferring it to the slaves, that will be problematic.
>
> Finally, there is an open pull request
> <https://github.com/apache/spark/pull/2554> related to IPython that may be
> relevant, though I haven’t looked at it too closely.
>
> Nick
>
>
>
> On Sat, Sep 27, 2014 at 7:33 PM, Andy Davidson <a...@santacruzintegration.com>
> wrote:
>> Hi
>>
>> I am having a heck of time trying to get python to work correctly on my
>> cluster created using the spark-ec2 script
>>
>> The following link was really helpful
>> https://issues.apache.org/jira/browse/SPARK-922
>>
>>
>> I am still running into problem with matplotlib. (it works fine on my mac). I
>> can not figure out how to get libagg, freetype, or Qhull dependencies
>> installed.
>>
>> Has anyone else run into this problem?
>>
>> Thanks
>>
>> Andy
>>
>> sudo yum install freetype-devel
>>
>> sudo yum install libpng-devel
>>
>> sudo pip2.7 install six
>>
>> sudo pip2.7 install python-dateutil
>>
>> sudo pip2.7 install pyparsing
>>
>> sudo pip2.7 install pycxx
>>
>>
>>
>> sudo pip2.7 install matplotlib
>>
>>
>> ec2-user@ip-172-31-15-87 ~]$ sudo pip2.7 install matplotlib
>>
>> Downloading/unpacking matplotlib
>>
>> Downloading matplotlib-1.4.0.tar.gz (51.2MB): 51.2MB downloaded
>>
>> Running setup.py (path:/tmp/pip_build_root/matplotlib/setup.py) egg_info
>> for package matplotlib
>>
>>
>> ============================================================================
>>
>> Edit setup.cfg to change the build options
>>
>>
>>
>> BUILDING MATPLOTLIB
>>
>> matplotlib: yes [1.4.0]
>>
>> python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) [GCC
>>
>> 4.8.2 20140120 (Red Hat 4.8.2-16)]]
>>
>> platform: yes [linux2]
>>
>>
>>
>> REQUIRED DEPENDENCIES AND EXTENSIONS
>>
>> numpy: yes [version 1.9.0]
>>
>> six: yes [using six version 1.8.0]
>>
>> dateutil: yes [using dateutil version 2.2]
>>
>> tornado: yes [using tornado version 4.0.2]
>>
>> pyparsing: yes [using pyparsing version 2.0.2]
>>
>> pycxx: yes [Couldn't import. Using local copy.]
>>
>> libagg: yes [pkg-config information for 'libagg' could
>> not
>>
>> be found. Using local copy.]
>>
>> freetype: no [Requires freetype2 2.4 or later. Found
>>
>> 2.3.11.]
>>
>> png: yes [version 1.2.49]
>>
>> qhull: yes [pkg-config information for 'qhull' could not
>> be
>>
>> found. Using local copy.]
>>
>>
>>
>> OPTIONAL SUBPACKAGES
>>
>> sample_data: yes [installing]
>>
>> toolkits: yes [installing]
>>
>> tests: yes [using nose version 1.3.4 / mock is required
>> to
>>
>> run the matplotlib test suite. pip/easy_install
>> may
>>
>> attempt to install it after matplotlib.]
>>
>> toolkits_tests: yes [using nose version 1.3.4 / mock is required
>> to
>>
>> run the matplotlib test suite. pip/easy_install
>> may
>>
>> attempt to install it after matplotlib.]
>>
>>
>>
>> OPTIONAL BACKEND EXTENSIONS
>>
>> macosx: no [Mac OS-X only]
>>
>> qt5agg: no [PyQt5 not found]
>>
>> qt4agg: no [PyQt4 not found]
>>
>> pyside: no [PySide not found]
>>
>> gtk3agg: no [Requires pygobject to be installed.]
>>
>> gtk3cairo: no [Requires cairocffi or pycairo to be
>> installed.]
>>
>> gtkagg: no [Requires pygtk]
>>
>> tkagg: no [TKAgg requires Tkinter.]
>>
>> wxagg: no [requires wxPython]
>>
>> gtk: no [Requires pygtk]
>>
>> agg: yes [installing]
>>
>> cairo: no [cairocffi or pycairo not found]
>>
>> windowing: no [Microsoft Windows only]
>>
>>
>>
>> OPTIONAL LATEX DEPENDENCIES
>>
>> dvipng: no
>>
>> ghostscript: yes [version 8.70]
>>
>> latex: yes [version 3.141592]
>>
>> pdftops: no
>>
>>
>>
>>
>> ============================================================================
>>
>> * The following required packages can not be
>> built:
>>
>> * freetype
>>
>> Complete output from command python setup.py egg_info:
>>
>>
>> ============================================================================
>>
>>
>>
>> Edit setup.cfg to change the build options
>>
>>
>>
>>
>>
>>
>>
>> BUILDING MATPLOTLIB
>>
>>
>>
>> matplotlib: yes [1.4.0]
>>
>>
>>
>> python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) [GCC
>>
>>
>>
>> 4.8.2 20140120 (Red Hat 4.8.2-16)]]
>>
>>
>>
>> platform: yes [linux2]
>>
>>
>>
>>
>>
>>
>>
>> REQUIRED DEPENDENCIES AND EXTENSIONS
>>
>>
>>
>> numpy: yes [version 1.9.0]
>>
>>
>>
>> six: yes [using six version 1.8.0]
>>
>>
>>
>> dateutil: yes [using dateutil version 2.2]
>>
>>
>>
>> tornado: yes [using tornado version 4.0.2]
>>
>>
>>
>> pyparsing: yes [using pyparsing version 2.0.2]
>>
>>
>>
>> pycxx: yes [Couldn't import. Using local copy.]
>>
>>
>>
>> libagg: yes [pkg-config information for 'libagg' could not
>>
>>
>>
>> be found. Using local copy.]
>>
>>
>>
>> freetype: no [Requires freetype2 2.4 or later. Found
>>
>>
>>
>> 2.3.11.]
>>
>>
>>
>> png: yes [version 1.2.49]
>>
>>
>>
>> qhull: yes [pkg-config information for 'qhull' could not be
>>
>>
>>
>> found. Using local copy.]
>>
>>
>>
>>
>>
>>
>>
>> OPTIONAL SUBPACKAGES
>>
>>
>>
>> sample_data: yes [installing]
>>
>>
>>
>> toolkits: yes [installing]
>>
>>
>>
>> tests: yes [using nose version 1.3.4 / mock is required to
>>
>>
>>
>> run the matplotlib test suite. pip/easy_install may
>>
>>
>>
>> attempt to install it after matplotlib.]
>>
>>
>>
>> toolkits_tests: yes [using nose version 1.3.4 / mock is required to
>>
>>
>>
>> run the matplotlib test suite. pip/easy_install may
>>
>>
>>
>> attempt to install it after matplotlib.]
>>
>>
>>
>>
>>
>>
>>
>> OPTIONAL BACKEND EXTENSIONS
>>
>>
>>
>> macosx: no [Mac OS-X only]
>>
>>
>>
>> qt5agg: no [PyQt5 not found]
>>
>>
>>
>> qt4agg: no [PyQt4 not found]
>>
>>
>>
>> pyside: no [PySide not found]
>>
>>
>>
>> gtk3agg: no [Requires pygobject to be installed.]
>>
>>
>>
>> gtk3cairo: no [Requires cairocffi or pycairo to be installed.]
>>
>>
>>
>> gtkagg: no [Requires pygtk]
>>
>>
>>
>> tkagg: no [TKAgg requires Tkinter.]
>>
>>
>>
>> wxagg: no [requires wxPython]
>>
>>
>>
>> gtk: no [Requires pygtk]
>>
>>
>>
>> agg: yes [installing]
>>
>>
>>
>> cairo: no [cairocffi or pycairo not found]
>>
>>
>>
>> windowing: no [Microsoft Windows only]
>>
>>
>>
>>
>>
>>
>>
>> OPTIONAL LATEX DEPENDENCIES
>>
>>
>>
>> dvipng: no
>>
>>
>>
>> ghostscript: yes [version 8.70]
>>
>>
>>
>> latex: yes [version 3.141592]
>>
>>
>>
>> pdftops: no
>>
>>
>>
>>
>>
>>
>>
>> ============================================================================
>>
>>
>>
>> * The following required packages can not be built:
>>
>>
>>
>> * freetype
>>
>>
>>
>> ----------------------------------------
>>
>> Cleaning up...
>>
>> Command python setup.py egg_info failed with error code 1 in
>> /tmp/pip_build_root/matplotlib
>>
>> Storing debug log for failure in /root/.pip/pip.log
>>
>> ec2-user@ip-172-31-15-87 ~]$
>>
>>
>
