Can you first confirm that the regular PySpark shell works on your cluster? Without upgrading to 2.7. That is, you log on to your master using spark-ec2 login and run bin/pyspark successfully without any special flags.
And as far as I can tell, you should be able to use IPython at 2.6, so I’d next confirm that that is working before throwing the 2.7 upgrade into the mix. Also, when upgrading or installing things, try doing so for all the nodes in your cluster using pssh. If you install stuff just on the master without somehow transferring it to the slaves, that will be problematic. Finally, there is an open pull request <https://github.com/apache/spark/pull/2554> related to IPython that may be relevant, though I haven’t looked at it too closely. Nick On Sat, Sep 27, 2014 at 7:33 PM, Andy Davidson < a...@santacruzintegration.com> wrote: > Hi > > I am having a heck of time trying to get python to work correctly on my > cluster created using the spark-ec2 script > > The following link was really helpful > https://issues.apache.org/jira/browse/SPARK-922 > > > I am still running into problem with matplotlib. (it works fine on my > mac). I can not figure out how to get libagg, freetype, or Qhull > dependencies installed. > > Has anyone else run into this problem? > > Thanks > > Andy > > sudo yum install freetype-devel > > sudo yum install libpng-devel > > sudo pip2.7 install six > > sudo pip2.7 install python-dateutil > > sudo pip2.7 install pyparsing > > sudo pip2.7 install pycxx > > > sudo pip2.7 install matplotlib > > ec2-user@ip-172-31-15-87 ~]$ sudo pip2.7 install matplotlib > > Downloading/unpacking matplotlib > > Downloading matplotlib-1.4.0.tar.gz (51.2MB): 51.2MB downloaded > > Running setup.py (path:/tmp/pip_build_root/matplotlib/setup.py) egg_info > for package matplotlib > > > ============================================================================ > > Edit setup.cfg to change the build options > > > > BUILDING MATPLOTLIB > > matplotlib: yes [1.4.0] > > python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) > [GCC > > 4.8.2 20140120 (Red Hat 4.8.2-16)]] > > platform: yes [linux2] > > > > REQUIRED DEPENDENCIES AND EXTENSIONS > > numpy: yes [version 1.9.0] > > six: yes [using six version 1.8.0] > > dateutil: yes [using dateutil version 2.2] > > tornado: yes [using tornado version 4.0.2] > > pyparsing: yes [using pyparsing version 2.0.2] > > pycxx: yes [Couldn't import. Using local copy.] > > libagg: yes [pkg-config information for 'libagg' could > not > > be found. Using local copy.] > > freetype: no [Requires freetype2 2.4 or later. Found > > 2.3.11.] > > png: yes [version 1.2.49] > > qhull: yes [pkg-config information for 'qhull' could > not be > > found. Using local copy.] > > > > OPTIONAL SUBPACKAGES > > sample_data: yes [installing] > > toolkits: yes [installing] > > tests: yes [using nose version 1.3.4 / mock is > required to > > run the matplotlib test suite. > pip/easy_install may > > attempt to install it after matplotlib.] > > toolkits_tests: yes [using nose version 1.3.4 / mock is > required to > > run the matplotlib test suite. > pip/easy_install may > > attempt to install it after matplotlib.] > > > > OPTIONAL BACKEND EXTENSIONS > > macosx: no [Mac OS-X only] > > qt5agg: no [PyQt5 not found] > > qt4agg: no [PyQt4 not found] > > pyside: no [PySide not found] > > gtk3agg: no [Requires pygobject to be installed.] > > gtk3cairo: no [Requires cairocffi or pycairo to be > installed.] > > gtkagg: no [Requires pygtk] > > tkagg: no [TKAgg requires Tkinter.] > > wxagg: no [requires wxPython] > > gtk: no [Requires pygtk] > > agg: yes [installing] > > cairo: no [cairocffi or pycairo not found] > > windowing: no [Microsoft Windows only] > > > > OPTIONAL LATEX DEPENDENCIES > > dvipng: no > > ghostscript: yes [version 8.70] > > latex: yes [version 3.141592] > > pdftops: no > > > > > ============================================================================ > > * The following required packages can not be > built: > > * freetype > > Complete output from command python setup.py egg_info: > > > ============================================================================ > > > Edit setup.cfg to change the build options > > > > > BUILDING MATPLOTLIB > > > matplotlib: yes [1.4.0] > > > python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) [GCC > > > 4.8.2 20140120 (Red Hat 4.8.2-16)]] > > > platform: yes [linux2] > > > > > REQUIRED DEPENDENCIES AND EXTENSIONS > > > numpy: yes [version 1.9.0] > > > six: yes [using six version 1.8.0] > > > dateutil: yes [using dateutil version 2.2] > > > tornado: yes [using tornado version 4.0.2] > > > pyparsing: yes [using pyparsing version 2.0.2] > > > pycxx: yes [Couldn't import. Using local copy.] > > > libagg: yes [pkg-config information for 'libagg' could not > > > be found. Using local copy.] > > > freetype: no [Requires freetype2 2.4 or later. Found > > > 2.3.11.] > > > png: yes [version 1.2.49] > > > qhull: yes [pkg-config information for 'qhull' could not > be > > > found. Using local copy.] > > > > > OPTIONAL SUBPACKAGES > > > sample_data: yes [installing] > > > toolkits: yes [installing] > > > tests: yes [using nose version 1.3.4 / mock is required to > > > run the matplotlib test suite. pip/easy_install > may > > > attempt to install it after matplotlib.] > > > toolkits_tests: yes [using nose version 1.3.4 / mock is required to > > > run the matplotlib test suite. pip/easy_install > may > > > attempt to install it after matplotlib.] > > > > > OPTIONAL BACKEND EXTENSIONS > > > macosx: no [Mac OS-X only] > > > qt5agg: no [PyQt5 not found] > > > qt4agg: no [PyQt4 not found] > > > pyside: no [PySide not found] > > > gtk3agg: no [Requires pygobject to be installed.] > > > gtk3cairo: no [Requires cairocffi or pycairo to be > installed.] > > > gtkagg: no [Requires pygtk] > > > tkagg: no [TKAgg requires Tkinter.] > > > wxagg: no [requires wxPython] > > > gtk: no [Requires pygtk] > > > agg: yes [installing] > > > cairo: no [cairocffi or pycairo not found] > > > windowing: no [Microsoft Windows only] > > > > > OPTIONAL LATEX DEPENDENCIES > > > dvipng: no > > > ghostscript: yes [version 8.70] > > > latex: yes [version 3.141592] > > > pdftops: no > > > > > > ============================================================================ > > > * The following required packages can not be built: > > > * freetype > > > ---------------------------------------- > > Cleaning up... > > Command python setup.py egg_info failed with error code 1 in > /tmp/pip_build_root/matplotlib > > Storing debug log for failure in /root/.pip/pip.log > > ec2-user@ip-172-31-15-87 ~]$ > >