Hi Nicholas Yes out of the box PySpark works. My problem is I am using iPython note book and matlabplot is not found. It seems that out of the box the cluster has an old version of python and iPython notebook. It was suggested I upgrade iPython because the new version include matlabplot. This upgrade requires going to python 2.7. The python upgrade and iPython upgrade seemed to work how ever I am still getting my original problem
ERROR: Line magic function `%matplotlib` not found I also posted to the iPython-dev mail list. So far I have not found a solution. Maybe I’ll have to switch to a different graphing package Thanks Andy From: Nicholas Chammas <nicholas.cham...@gmail.com> Date: Saturday, September 27, 2014 at 4:49 PM To: Andrew Davidson <a...@santacruzintegration.com> Cc: "user@spark.apache.org" <user@spark.apache.org> Subject: Re: iPython notebook ec2 cluster matlabplot not found? > Can you first confirm that the regular PySpark shell works on your cluster? > Without upgrading to 2.7. That is, you log on to your master using spark-ec2 > login and run bin/pyspark successfully without any special flags. > > And as far as I can tell, you should be able to use IPython at 2.6, so I’d > next confirm that that is working before throwing the 2.7 upgrade into the > mix. > > Also, when upgrading or installing things, try doing so for all the nodes in > your cluster using pssh. If you install stuff just on the master without > somehow transferring it to the slaves, that will be problematic. > > Finally, there is an open pull request > <https://github.com/apache/spark/pull/2554> related to IPython that may be > relevant, though I haven’t looked at it too closely. > > Nick > > > > On Sat, Sep 27, 2014 at 7:33 PM, Andy Davidson <a...@santacruzintegration.com> > wrote: >> Hi >> >> I am having a heck of time trying to get python to work correctly on my >> cluster created using the spark-ec2 script >> >> The following link was really helpful >> https://issues.apache.org/jira/browse/SPARK-922 >> >> >> I am still running into problem with matplotlib. (it works fine on my mac). I >> can not figure out how to get libagg, freetype, or Qhull dependencies >> installed. >> >> Has anyone else run into this problem? >> >> Thanks >> >> Andy >> >> sudo yum install freetype-devel >> >> sudo yum install libpng-devel >> >> sudo pip2.7 install six >> >> sudo pip2.7 install python-dateutil >> >> sudo pip2.7 install pyparsing >> >> sudo pip2.7 install pycxx >> >> >> >> sudo pip2.7 install matplotlib >> >> >> ec2-user@ip-172-31-15-87 ~]$ sudo pip2.7 install matplotlib >> >> Downloading/unpacking matplotlib >> >> Downloading matplotlib-1.4.0.tar.gz (51.2MB): 51.2MB downloaded >> >> Running setup.py (path:/tmp/pip_build_root/matplotlib/setup.py) egg_info >> for package matplotlib >> >> >> ============================================================================ >> >> Edit setup.cfg to change the build options >> >> >> >> BUILDING MATPLOTLIB >> >> matplotlib: yes [1.4.0] >> >> python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) [GCC >> >> 4.8.2 20140120 (Red Hat 4.8.2-16)]] >> >> platform: yes [linux2] >> >> >> >> REQUIRED DEPENDENCIES AND EXTENSIONS >> >> numpy: yes [version 1.9.0] >> >> six: yes [using six version 1.8.0] >> >> dateutil: yes [using dateutil version 2.2] >> >> tornado: yes [using tornado version 4.0.2] >> >> pyparsing: yes [using pyparsing version 2.0.2] >> >> pycxx: yes [Couldn't import. Using local copy.] >> >> libagg: yes [pkg-config information for 'libagg' could >> not >> >> be found. Using local copy.] >> >> freetype: no [Requires freetype2 2.4 or later. Found >> >> 2.3.11.] >> >> png: yes [version 1.2.49] >> >> qhull: yes [pkg-config information for 'qhull' could not >> be >> >> found. Using local copy.] >> >> >> >> OPTIONAL SUBPACKAGES >> >> sample_data: yes [installing] >> >> toolkits: yes [installing] >> >> tests: yes [using nose version 1.3.4 / mock is required >> to >> >> run the matplotlib test suite. pip/easy_install >> may >> >> attempt to install it after matplotlib.] >> >> toolkits_tests: yes [using nose version 1.3.4 / mock is required >> to >> >> run the matplotlib test suite. pip/easy_install >> may >> >> attempt to install it after matplotlib.] >> >> >> >> OPTIONAL BACKEND EXTENSIONS >> >> macosx: no [Mac OS-X only] >> >> qt5agg: no [PyQt5 not found] >> >> qt4agg: no [PyQt4 not found] >> >> pyside: no [PySide not found] >> >> gtk3agg: no [Requires pygobject to be installed.] >> >> gtk3cairo: no [Requires cairocffi or pycairo to be >> installed.] >> >> gtkagg: no [Requires pygtk] >> >> tkagg: no [TKAgg requires Tkinter.] >> >> wxagg: no [requires wxPython] >> >> gtk: no [Requires pygtk] >> >> agg: yes [installing] >> >> cairo: no [cairocffi or pycairo not found] >> >> windowing: no [Microsoft Windows only] >> >> >> >> OPTIONAL LATEX DEPENDENCIES >> >> dvipng: no >> >> ghostscript: yes [version 8.70] >> >> latex: yes [version 3.141592] >> >> pdftops: no >> >> >> >> >> ============================================================================ >> >> * The following required packages can not be >> built: >> >> * freetype >> >> Complete output from command python setup.py egg_info: >> >> >> ============================================================================ >> >> >> >> Edit setup.cfg to change the build options >> >> >> >> >> >> >> >> BUILDING MATPLOTLIB >> >> >> >> matplotlib: yes [1.4.0] >> >> >> >> python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) [GCC >> >> >> >> 4.8.2 20140120 (Red Hat 4.8.2-16)]] >> >> >> >> platform: yes [linux2] >> >> >> >> >> >> >> >> REQUIRED DEPENDENCIES AND EXTENSIONS >> >> >> >> numpy: yes [version 1.9.0] >> >> >> >> six: yes [using six version 1.8.0] >> >> >> >> dateutil: yes [using dateutil version 2.2] >> >> >> >> tornado: yes [using tornado version 4.0.2] >> >> >> >> pyparsing: yes [using pyparsing version 2.0.2] >> >> >> >> pycxx: yes [Couldn't import. Using local copy.] >> >> >> >> libagg: yes [pkg-config information for 'libagg' could not >> >> >> >> be found. Using local copy.] >> >> >> >> freetype: no [Requires freetype2 2.4 or later. Found >> >> >> >> 2.3.11.] >> >> >> >> png: yes [version 1.2.49] >> >> >> >> qhull: yes [pkg-config information for 'qhull' could not be >> >> >> >> found. Using local copy.] >> >> >> >> >> >> >> >> OPTIONAL SUBPACKAGES >> >> >> >> sample_data: yes [installing] >> >> >> >> toolkits: yes [installing] >> >> >> >> tests: yes [using nose version 1.3.4 / mock is required to >> >> >> >> run the matplotlib test suite. pip/easy_install may >> >> >> >> attempt to install it after matplotlib.] >> >> >> >> toolkits_tests: yes [using nose version 1.3.4 / mock is required to >> >> >> >> run the matplotlib test suite. pip/easy_install may >> >> >> >> attempt to install it after matplotlib.] >> >> >> >> >> >> >> >> OPTIONAL BACKEND EXTENSIONS >> >> >> >> macosx: no [Mac OS-X only] >> >> >> >> qt5agg: no [PyQt5 not found] >> >> >> >> qt4agg: no [PyQt4 not found] >> >> >> >> pyside: no [PySide not found] >> >> >> >> gtk3agg: no [Requires pygobject to be installed.] >> >> >> >> gtk3cairo: no [Requires cairocffi or pycairo to be installed.] >> >> >> >> gtkagg: no [Requires pygtk] >> >> >> >> tkagg: no [TKAgg requires Tkinter.] >> >> >> >> wxagg: no [requires wxPython] >> >> >> >> gtk: no [Requires pygtk] >> >> >> >> agg: yes [installing] >> >> >> >> cairo: no [cairocffi or pycairo not found] >> >> >> >> windowing: no [Microsoft Windows only] >> >> >> >> >> >> >> >> OPTIONAL LATEX DEPENDENCIES >> >> >> >> dvipng: no >> >> >> >> ghostscript: yes [version 8.70] >> >> >> >> latex: yes [version 3.141592] >> >> >> >> pdftops: no >> >> >> >> >> >> >> >> ============================================================================ >> >> >> >> * The following required packages can not be built: >> >> >> >> * freetype >> >> >> >> ---------------------------------------- >> >> Cleaning up... >> >> Command python setup.py egg_info failed with error code 1 in >> /tmp/pip_build_root/matplotlib >> >> Storing debug log for failure in /root/.pip/pip.log >> >> ec2-user@ip-172-31-15-87 ~]$ >> >> >
