Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
about to send mail. On Thu, Mar 14, 2019 at 1:00 PM Marcelo Vanzin wrote: > Not sure if anyone else is having the same issues, but jenkins seems > to be in a weird state. > > I can't connect to the web UI, and it doesn't seem to be responding to > test requests. > > On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > > > upgrade completed, jenkins building again... master PR merged, waiting > for the 2.4.1 PR to launch the k8s integration tests. > > > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > >> > >> okie dokie! the time approacheth! > >> > >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > upgrade to take more than 15-20 mins, following which i will re-enable > builds. > >> > >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp > wrote: > >>> > >>> ok awesome. let's shoot for 3pm PST. > >>> > >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin > wrote: > > On Wed, Mar 13, 2019 at 11:53 AM shane knapp > wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin < > van...@cloudera.com> wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master > client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge that > PR once it's done. Then I'll take care of backporting it to 2.4 and > make sure it passes the integration tests. > > -- > Marcelo > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Marcelo > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
Not sure if anyone else is having the same issues, but jenkins seems to be in a weird state. I can't connect to the web UI, and it doesn't seem to be responding to test requests. On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > upgrade completed, jenkins building again... master PR merged, waiting for > the 2.4.1 PR to launch the k8s integration tests. > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: >> >> okie dokie! the time approacheth! >> >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the upgrade >> to take more than 15-20 mins, following which i will re-enable builds. >> >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: >>> >>> ok awesome. let's shoot for 3pm PST. >>> >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: >> >> Do the upgraded minikube/k8s versions break the current master client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: since the master PR is good to go pending the minikube upgrade, let's try to synchronize things. Set a time to do the minikube upgrade this PM, if that works for you, and I'll merge that PR once it's done. Then I'll take care of backporting it to 2.4 and make sure it passes the integration tests. -- Marcelo >>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
i think we should. the version of k8s we were testing against before was v1.10.x, and we are currently testing master *and* 2.4.1 against v1.13.1. given that we're a couple of weeks away from v.1.14.x i feel that this is a safe and prudent thing to include. On Thu, Mar 14, 2019 at 10:57 AM DB Tsai wrote: > Since rc8 was already cut without the k8s client upgrade; the build is > ready to vote, and including k8s client upgrade in 2.4.1 implies that > we will drop the old-but-not-that-old > K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? > > Thanks. > > Sincerely, > > DB Tsai > -- > Web: https://www.dbtsai.com > PGP Key ID: 42E5B25A8F7A82C1 > > On Thu, Mar 14, 2019 at 9:48 AM shane knapp wrote: > > > > thanks everyone, both PRs are merged. :) > > > > On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > >> > >> btw, let's wait and see if the non-k8s PRB tests pass before merging > https://github.com/apache/spark/pull/23993 in to 2.4.1 > >> > >> On Wed, Mar 13, 2019 at 3:42 PM shane knapp > wrote: > >>> > >>> 2.4.1 k8s integration test passed: > >>> > >>> > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > >>> > >>> thanks everyone! :) > >>> > >>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp > wrote: > > 2.4.1 integration tests running: > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > > On Wed, Mar 13, 2019 at 3:15 PM shane knapp > wrote: > > > > upgrade completed, jenkins building again... master PR merged, > waiting for the 2.4.1 PR to launch the k8s integration tests. > > > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp > wrote: > >> > >> okie dokie! the time approacheth! > >> > >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect > the upgrade to take more than 15-20 mins, following which i will re-enable > builds. > >> > >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp > wrote: > >>> > >>> ok awesome. let's shoot for 3pm PST. > >>> > >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin < > van...@cloudera.com> wrote: > > On Wed, Mar 13, 2019 at 11:53 AM shane knapp > wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin < > van...@cloudera.com> wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master > client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the > minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge > that > PR once it's done. Then I'll take care of backporting it to 2.4 > and > make sure it passes the integration tests. > > -- > Marcelo > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
IMO we can vote on rc8; that upgrade should not block it. If it fails for some other reason, then rc9 will have it. Supported versions in k8s-land are really weird and don't match well to our release schedule. There isn't a good solution to this at the moment... good thing k8s support is still marked as experimental. :-) On Thu, Mar 14, 2019 at 10:56 AM DB Tsai wrote: > > Since rc8 was already cut without the k8s client upgrade; the build is > ready to vote, and including k8s client upgrade in 2.4.1 implies that > we will drop the old-but-not-that-old > K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? > > Thanks. > > Sincerely, > > DB Tsai > -- > Web: https://www.dbtsai.com > PGP Key ID: 42E5B25A8F7A82C1 > > On Thu, Mar 14, 2019 at 9:48 AM shane knapp wrote: > > > > thanks everyone, both PRs are merged. :) > > > > On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > >> > >> btw, let's wait and see if the non-k8s PRB tests pass before merging > >> https://github.com/apache/spark/pull/23993 in to 2.4.1 > >> > >> On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: > >>> > >>> 2.4.1 k8s integration test passed: > >>> > >>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > >>> > >>> thanks everyone! :) > >>> > >>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: > > 2.4.1 integration tests running: > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > > On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > > > upgrade completed, jenkins building again... master PR merged, waiting > > for the 2.4.1 PR to launch the k8s integration tests. > > > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > >> > >> okie dokie! the time approacheth! > >> > >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > >> upgrade to take more than 15-20 mins, following which i will re-enable > >> builds. > >> > >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp > >> wrote: > >>> > >>> ok awesome. let's shoot for 3pm PST. > >>> > >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin > >>> wrote: > > On Wed, Mar 13, 2019 at 11:53 AM shane knapp > wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > > wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master > >> client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge that > PR once it's done. Then I'll take care of backporting it to 2.4 and > make sure it passes the integration tests. > > -- > Marcelo > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
Since rc8 was already cut without the k8s client upgrade; the build is ready to vote, and including k8s client upgrade in 2.4.1 implies that we will drop the old-but-not-that-old K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Thu, Mar 14, 2019 at 9:48 AM shane knapp wrote: > > thanks everyone, both PRs are merged. :) > > On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: >> >> btw, let's wait and see if the non-k8s PRB tests pass before merging >> https://github.com/apache/spark/pull/23993 in to 2.4.1 >> >> On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: >>> >>> 2.4.1 k8s integration test passed: >>> >>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ >>> >>> thanks everyone! :) >>> >>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: 2.4.1 integration tests running: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > upgrade completed, jenkins building again... master PR merged, waiting > for the 2.4.1 PR to launch the k8s integration tests. > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: >> >> okie dokie! the time approacheth! >> >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the >> upgrade to take more than 15-20 mins, following which i will re-enable >> builds. >> >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: >>> >>> ok awesome. let's shoot for 3pm PST. >>> >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin >>> wrote: On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: >> >> Do the upgraded minikube/k8s versions break the current master >> client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: since the master PR is good to go pending the minikube upgrade, let's try to synchronize things. Set a time to do the minikube upgrade this PM, if that works for you, and I'll merge that PR once it's done. Then I'll take care of backporting it to 2.4 and make sure it passes the integration tests. -- Marcelo >>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
thanks everyone, both PRs are merged. :) On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > btw, let's wait and see if the non-k8s PRB tests pass before merging > https://github.com/apache/spark/pull/23993 in to 2.4.1 > > On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: > >> 2.4.1 k8s integration test passed: >> >> >> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ >> >> thanks everyone! :) >> >> On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: >> >>> 2.4.1 integration tests running: >>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ >>> >>> On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: >>> upgrade completed, jenkins building again... master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests. On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > okie dokie! the time approacheth! > > i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > upgrade to take more than 15-20 mins, following which i will re-enable > builds. > > On Wed, Mar 13, 2019 at 12:17 PM shane knapp > wrote: > >> ok awesome. let's shoot for 3pm PST. >> >> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin >> wrote: >> >>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp >>> wrote: >>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin < >>> van...@cloudera.com> wrote: >>> >> >>> >> Do the upgraded minikube/k8s versions break the current master >>> client >>> >> version too? >>> >> >>> > yes. >>> >>> Ah, so that part kinda sucks. >>> >>> Let's do this: since the master PR is good to go pending the minikube >>> upgrade, let's try to synchronize things. Set a time to do the >>> minikube upgrade this PM, if that works for you, and I'll merge that >>> PR once it's done. Then I'll take care of backporting it to 2.4 and >>> make sure it passes the integration tests. >>> >>> -- >>> Marcelo >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
btw, let's wait and see if the non-k8s PRB tests pass before merging https://github.com/apache/spark/pull/23993 in to 2.4.1 On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: > 2.4.1 k8s integration test passed: > > > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > > thanks everyone! :) > > On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: > >> 2.4.1 integration tests running: >> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ >> >> On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: >> >>> upgrade completed, jenkins building again... master PR merged, waiting >>> for the 2.4.1 PR to launch the k8s integration tests. >>> >>> On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: >>> okie dokie! the time approacheth! i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds. On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: > ok awesome. let's shoot for 3pm PST. > > On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin > wrote: > >> On Wed, Mar 13, 2019 at 11:53 AM shane knapp >> wrote: >> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin < >> van...@cloudera.com> wrote: >> >> >> >> Do the upgraded minikube/k8s versions break the current master >> client >> >> version too? >> >> >> > yes. >> >> Ah, so that part kinda sucks. >> >> Let's do this: since the master PR is good to go pending the minikube >> upgrade, let's try to synchronize things. Set a time to do the >> minikube upgrade this PM, if that works for you, and I'll merge that >> PR once it's done. Then I'll take care of backporting it to 2.4 and >> make sure it passes the integration tests. >> >> -- >> Marcelo >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
2.4.1 k8s integration test passed: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ thanks everyone! :) On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: > 2.4.1 integration tests running: > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > > On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > >> upgrade completed, jenkins building again... master PR merged, waiting >> for the 2.4.1 PR to launch the k8s integration tests. >> >> On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: >> >>> okie dokie! the time approacheth! >>> >>> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the >>> upgrade to take more than 15-20 mins, following which i will re-enable >>> builds. >>> >>> On Wed, Mar 13, 2019 at 12:17 PM shane knapp >>> wrote: >>> ok awesome. let's shoot for 3pm PST. On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: > On Wed, Mar 13, 2019 at 11:53 AM shane knapp > wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master > client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge that > PR once it's done. Then I'll take care of backporting it to 2.4 and > make sure it passes the integration tests. > > -- > Marcelo > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
2.4.1 integration tests running: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > upgrade completed, jenkins building again... master PR merged, waiting > for the 2.4.1 PR to launch the k8s integration tests. > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > >> okie dokie! the time approacheth! >> >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the >> upgrade to take more than 15-20 mins, following which i will re-enable >> builds. >> >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: >> >>> ok awesome. let's shoot for 3pm PST. >>> >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin >>> wrote: >>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin wrote: >> >> Do the upgraded minikube/k8s versions break the current master client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: since the master PR is good to go pending the minikube upgrade, let's try to synchronize things. Set a time to do the minikube upgrade this PM, if that works for you, and I'll merge that PR once it's done. Then I'll take care of backporting it to 2.4 and make sure it passes the integration tests. -- Marcelo >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
upgrade completed, jenkins building again... master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests. On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > okie dokie! the time approacheth! > > i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > upgrade to take more than 15-20 mins, following which i will re-enable > builds. > > On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: > >> ok awesome. let's shoot for 3pm PST. >> >> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin >> wrote: >> >>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp >>> wrote: >>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin >>> wrote: >>> >> >>> >> Do the upgraded minikube/k8s versions break the current master client >>> >> version too? >>> >> >>> > yes. >>> >>> Ah, so that part kinda sucks. >>> >>> Let's do this: since the master PR is good to go pending the minikube >>> upgrade, let's try to synchronize things. Set a time to do the >>> minikube upgrade this PM, if that works for you, and I'll merge that >>> PR once it's done. Then I'll take care of backporting it to 2.4 and >>> make sure it passes the integration tests. >>> >>> -- >>> Marcelo >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
okie dokie! the time approacheth! i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds. On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: > ok awesome. let's shoot for 3pm PST. > > On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin > wrote: > >> On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: >> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin >> wrote: >> >> >> >> Do the upgraded minikube/k8s versions break the current master client >> >> version too? >> >> >> > yes. >> >> Ah, so that part kinda sucks. >> >> Let's do this: since the master PR is good to go pending the minikube >> upgrade, let's try to synchronize things. Set a time to do the >> minikube upgrade this PM, if that works for you, and I'll merge that >> PR once it's done. Then I'll take care of backporting it to 2.4 and >> make sure it passes the integration tests. >> >> -- >> Marcelo >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
Sounds good. On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: > > ok awesome. let's shoot for 3pm PST. > > On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: >> >> On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: >> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin >> > wrote: >> >> >> >> Do the upgraded minikube/k8s versions break the current master client >> >> version too? >> >> >> > yes. >> >> Ah, so that part kinda sucks. >> >> Let's do this: since the master PR is good to go pending the minikube >> upgrade, let's try to synchronize things. Set a time to do the >> minikube upgrade this PM, if that works for you, and I'll merge that >> PR once it's done. Then I'll take care of backporting it to 2.4 and >> make sure it passes the integration tests. >> >> -- >> Marcelo > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
ok awesome. let's shoot for 3pm PST. On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: > On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge that > PR once it's done. Then I'll take care of backporting it to 2.4 and > make sure it passes the integration tests. > > -- > Marcelo > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
I'm OK with this take. The problem with back-porting the client update to 2.4.x at all is that it drops support for some old-but-not-that-old K8S versions, which feels surprising in a maintenance release. That said, maybe it's OK, and a little more OK for a 2.4.2 in several months' time. On Wed, Mar 13, 2019 at 1:49 PM Marcelo Vanzin wrote: > > Do the upgraded minikube/k8s versions break the current master client > version too? > > I'm not super concerned about 2.4 integration tests being broken for a > little bit. It's very uncommon for new PRs to be open against > branch-2.4 that would affect k8s. > > But I really don't want master to break. So if we can upgrade minikube > first, even if that breaks k8s integration tests on branch-2.4 for a > little bit, that would be optimal IMO. > > On Wed, Mar 13, 2019 at 11:26 AM shane knapp wrote: > > > > hey everyone... i wanted to break this discussion out of the mega-threads > > for the 2.4.1 RC candidates. > > > > the TL;DR is that we've been trying to update the k8s client libs to > > something much more modern. however, for us to do this, we need to update > > our very old k8s and minikube versions. > > > > the problem here lies in the fact that if we update the client libs on > > master, but not the 2.4 branch, then the 2.4 branch k8s integration tests > > will fail if we update our backend minikube/k8s versions. > > > > i've done all of the testing locally for the new k8s client libs, and am > > ready to pull the trigger on the infrastructure upgrade (which will take > > all of ~15 mins). > > > > for this to happen, two PRs will need to be merged... one for 2.4.1 and > > one for master. > > > > is there a chance that we can get > > https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 release? > > this will also require https://github.com/apache/spark/pull/24002 (for > > master) to be merged simultaneously. > > > > both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm > > not entirely sure why). > > > > here's the primary jira we're using to track this upgrade: > > https://issues.apache.org/jira/browse/SPARK-26742 > > > > thanks in advance, > > > > shane > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin wrote: >> >> Do the upgraded minikube/k8s versions break the current master client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: since the master PR is good to go pending the minikube upgrade, let's try to synchronize things. Set a time to do the minikube upgrade this PM, if that works for you, and I'll merge that PR once it's done. Then I'll take care of backporting it to 2.4 and make sure it passes the integration tests. -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin wrote: > Do the upgraded minikube/k8s versions break the current master client > version too? > > yes. > I'm not super concerned about 2.4 integration tests being broken for a > little bit. It's very uncommon for new PRs to be open against > branch-2.4 that would affect k8s. > > ok. if the 2.4.1 PR is merged at about the same time as the one for master, we won't have to worry about the 2.4.x tests failing. But I really don't want master to break. So if we can upgrade minikube > first, even if that breaks k8s integration tests on branch-2.4 for a > little bit, that would be optimal IMO. > > i have everything staged on the affected jenkins workers and can do the infra upgrade really quickly... shane > On Wed, Mar 13, 2019 at 11:26 AM shane knapp wrote: > > > > hey everyone... i wanted to break this discussion out of the > mega-threads for the 2.4.1 RC candidates. > > > > the TL;DR is that we've been trying to update the k8s client libs to > something much more modern. however, for us to do this, we need to update > our very old k8s and minikube versions. > > > > the problem here lies in the fact that if we update the client libs on > master, but not the 2.4 branch, then the 2.4 branch k8s integration tests > will fail if we update our backend minikube/k8s versions. > > > > i've done all of the testing locally for the new k8s client libs, and am > ready to pull the trigger on the infrastructure upgrade (which will take > all of ~15 mins). > > > > for this to happen, two PRs will need to be merged... one for 2.4.1 and > one for master. > > > > is there a chance that we can get > https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 > release? this will also require > https://github.com/apache/spark/pull/24002 (for master) to be merged > simultaneously. > > > > both of those PRs are ready to go (tho 23993 was closed w/o merge and > i'm not entirely sure why). > > > > here's the primary jira we're using to track this upgrade: > > https://issues.apache.org/jira/browse/SPARK-26742 > > > > thanks in advance, > > > > shane > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Marcelo > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
Do the upgraded minikube/k8s versions break the current master client version too? I'm not super concerned about 2.4 integration tests being broken for a little bit. It's very uncommon for new PRs to be open against branch-2.4 that would affect k8s. But I really don't want master to break. So if we can upgrade minikube first, even if that breaks k8s integration tests on branch-2.4 for a little bit, that would be optimal IMO. On Wed, Mar 13, 2019 at 11:26 AM shane knapp wrote: > > hey everyone... i wanted to break this discussion out of the mega-threads > for the 2.4.1 RC candidates. > > the TL;DR is that we've been trying to update the k8s client libs to > something much more modern. however, for us to do this, we need to update > our very old k8s and minikube versions. > > the problem here lies in the fact that if we update the client libs on > master, but not the 2.4 branch, then the 2.4 branch k8s integration tests > will fail if we update our backend minikube/k8s versions. > > i've done all of the testing locally for the new k8s client libs, and am > ready to pull the trigger on the infrastructure upgrade (which will take all > of ~15 mins). > > for this to happen, two PRs will need to be merged... one for 2.4.1 and one > for master. > > is there a chance that we can get https://github.com/apache/spark/pull/23993 > merged in for the 2.4.1 release? this will also require > https://github.com/apache/spark/pull/24002 (for master) to be merged > simultaneously. > > both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm not > entirely sure why). > > here's the primary jira we're using to track this upgrade: > https://issues.apache.org/jira/browse/SPARK-26742 > > thanks in advance, > > shane > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
[discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
hey everyone... i wanted to break this discussion out of the mega-threads for the 2.4.1 RC candidates. the TL;DR is that we've been trying to update the k8s client libs to something much more modern. however, for us to do this, we need to update our very old k8s and minikube versions. the problem here lies in the fact that if we update the client libs on master, but not the 2.4 branch, then the 2.4 branch k8s integration tests will fail if we update our backend minikube/k8s versions. i've done all of the testing locally for the new k8s client libs, and am ready to pull the trigger on the infrastructure upgrade (which will take all of ~15 mins). for this to happen, two PRs will need to be merged... one for 2.4.1 and one for master. is there a chance that we can get https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 release? this will also require https://github.com/apache/spark/pull/24002 (for master) to be merged simultaneously. both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm not entirely sure why). here's the primary jira we're using to track this upgrade: https://issues.apache.org/jira/browse/SPARK-26742 thanks in advance, shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
