i'm going to punt on this until after the next spark 1.3 release (2-3 weeks?). since i'll be installing a bunch of other packages (including mongodb), i'd rather wait and be safe. :)
the full install list is forthcoming, and i'll update the spark infra wiki w/what's installed on the workers. shane On Mon, Feb 23, 2015 at 11:13 AM, shane knapp <skn...@berkeley.edu> wrote: > good morning, developers! > > TL;DR: > > i will be installing anaconda and setting it in the system PATH so that > your python will default to 2.7, as well as it taking over management of > all of the sci-py packages. this is potentially a big change, so i'll be > testing locally on my staging instance before deployment to the wide world. > > deployment is *tentatively* next monday, march 2nd. > > a little background: > > the jenkins test infra is currently (and happily) managed by a set of > tools that allow me to set up and deploy new workers, manage their packages > and make sure that all spark and research projects can happily and > successfully build. > > we're currently at the state where ~50 or so packages are installed and > configured on each worker. this is getting a little cumbersome, as the > package-to-build dep tree is getting pretty large. > > the biggest offender is the science-based python infrastructure. > everything is blindly installed w/yum and pip, so it's hard to control > *exactly* what version of any given library is as compared to what's on a > dev's laptop. > > the solution: > > anaconda (https://store.continuum.io/cshop/anaconda/)! everything is > centralized! i can manage specific versions much easier! > > what this means to you: > > * python 2.7 will be the default system python. > * 2.6 will still be installed and available (/usr/bin/python or > /usr/bin/python/2.6) > > what you need to do: > * install anaconda, have it update your PATH > * build locally and try to fix any bugs (for spark, this "should just > work") > * if you have problems, reach out to me and i'll see what i can do to > help. if we can't get your stuff running under python2.7, we can default > to 2.6 via a job config change. > > what i will be doing: > * setting up anaconda on my staging instance and spot-testing a lot of > builds before deployment > > please let me know if there are any issues/concerns... i'll be posting > updates this week and will let everyone know if there are any changes to > the Plan[tm]. > > your friendly devops engineer, > > shane >
