test builds started (on ubuntu). if these pass, i will feel comfortable performing the same installation/upgrade steps on the centos workers.
upgrade/installation commands: conda install python==3.5 conda install pyarrow=0.10.* -c conda-forge -n py3k pip install sphinx builds: https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/spark-master-test-sbt-hadoop-2.6-ubuntu-test/899/ https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/66/ hopefully python3.5 won't break anything! please hold for ~4.5 hours. ;) On Wed, Aug 8, 2018 at 11:57 AM, shane knapp <skn...@berkeley.edu> wrote: > i updated my staging/testing ubuntu server to python 3.5, successfully > installed pyarrow 0.10.0 via conda forge. however i'm getting a ton of > python package dep failures. > > i will see if we can get by w/o needing to wipe and recreate every > anaconda installation. please hold. > > On Wed, Aug 8, 2018 at 11:48 AM, shane knapp <skn...@berkeley.edu> wrote: > >> well... i've been running in to problems (aka dependency hell), and just >> hit a show-stopper: >> >> UnsatisfiableError: The following specifications were found to be in >> conflict: >> - pyarrow 0.10.* -> arrow-cpp 0.10.0.* -> python >=2.7,<2.8.0a0 >> - python 3.4* >> Use "conda info <package>" to see the dependencies for each package. >> >> yep, we're testing against python 3.4, and pyarrow 0.10.0 needs 3.5+ >> >> putting the workers back on-line until i figure out what to do next. >> >> shane >> >> On Wed, Aug 8, 2018 at 10:31 AM, shane knapp <skn...@berkeley.edu> wrote: >> >>> pyarrow 0.10.0 has been released, and this is important to be tested >>> against for the 2.4 release (esp due to memory leak problems, etc) >>> >>> https://issues.apache.org/jira/browse/SPARK-23874 >>> https://github.com/apache/spark/pull/21939 >>> https://issues.apache.org/jira/browse/ARROW-1973 >>> >>> i will be putting the centos workers in to quiet mode now, prepping the >>> upgrade and once the majority of existing builds are done, i'll kill any >>> outliers (and retrigger them later) and perform the upgrade. >>> >>> expect this to take ~3 hours, max. >>> >>> shane >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
