On Wed, Aug 15, 2018 at 12:45 PM, Reynold Xin <r...@databricks.com> wrote:
> What's the reason we don't want to do the OS updates right now? Is it due > to the unpredictability of potential issues that might happen and end up > delaying 2.4 release? > > that is exactly it... i haven't had a chance to test everything (esp docs, release builds, etc), and since there have been many changes pushed to various branches, some builds are failing in different ways on centos vs ubuntu. things are very, very fluid right now. we are SUPER close to being able to move the vast majority of builds to ubuntu, but even after than happens there will almost definitely be a non-zero amount of babysitting and debugging that needs to happen. all of that babysitting and debugging would potentially delay the 2.4 cut even more. in fact, i don't see us getting rid of all of the centos machines until EOY (see my above comment, re docs, release etc). these are the builds that will remain on centos for the near future: https://rise.cs.berkeley.edu/jenkins/label/spark-release/ https://rise.cs.berkeley.edu/jenkins/label/spark-packaging/ https://rise.cs.berkeley.edu/jenkins/label/spark-docs/ these all run on amp-jenkins-worker-01, which will remain untouched. HOWEVER: i'm 99%+ certain that i can port the PRB builder over to ubuntu w/a pared down python 3.5 installation. i have a build running now ( https://rise.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/74/) and if it behaves appropriately (meaning testing all the right things w/py35), i should be able to get two more non-critical centos build nodes reinstalled w/ubuntu and deployed by EOW. this will give us enough jenkins executors and allow PRB throughput to not decrease. if the dev community is ok w/taking this plunge, i will make it happen. shane (who wants everyone to remember that it's just little old me running this... not a team of people) ;) -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
