Scala type checking thread-safety issue, and global locks to resolve it
This is worth a look: https://github.com/apache/spark/pull/24085 Scala has a surprising thread-safety bug in the "<:<" operator that's used to check subtypes, which can lead to incorrect results in non-trivial situations. The fix on the table is to introduce a global lock to protect a lot of the Scala-related reflection code to resolve it. This may be the best we can do, but 'global lock' sounds ominous. Any thoughts on other ways to resolve it? I'm not sure there are. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [build system] jenkins wedged again, restarted service.
things are building again! On Thu, Mar 14, 2019 at 2:02 PM shane knapp wrote: > if performance continues to drop off like this, i will reboot the box soon. > > builds should be back to normal in a few minutes... sorry for the > inconvenience! > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
[build system] jenkins wedged again, restarted service.
if performance continues to drop off like this, i will reboot the box soon. builds should be back to normal in a few minutes... sorry for the inconvenience! -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
about to send mail. On Thu, Mar 14, 2019 at 1:00 PM Marcelo Vanzin wrote: > Not sure if anyone else is having the same issues, but jenkins seems > to be in a weird state. > > I can't connect to the web UI, and it doesn't seem to be responding to > test requests. > > On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > > > upgrade completed, jenkins building again... master PR merged, waiting > for the 2.4.1 PR to launch the k8s integration tests. > > > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > >> > >> okie dokie! the time approacheth! > >> > >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > upgrade to take more than 15-20 mins, following which i will re-enable > builds. > >> > >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp > wrote: > >>> > >>> ok awesome. let's shoot for 3pm PST. > >>> > >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin > wrote: > > On Wed, Mar 13, 2019 at 11:53 AM shane knapp > wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin < > van...@cloudera.com> wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master > client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge that > PR once it's done. Then I'll take care of backporting it to 2.4 and > make sure it passes the integration tests. > > -- > Marcelo > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Marcelo > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
Not sure if anyone else is having the same issues, but jenkins seems to be in a weird state. I can't connect to the web UI, and it doesn't seem to be responding to test requests. On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > upgrade completed, jenkins building again... master PR merged, waiting for > the 2.4.1 PR to launch the k8s integration tests. > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: >> >> okie dokie! the time approacheth! >> >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the upgrade >> to take more than 15-20 mins, following which i will re-enable builds. >> >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: >>> >>> ok awesome. let's shoot for 3pm PST. >>> >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: >> >> Do the upgraded minikube/k8s versions break the current master client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: since the master PR is good to go pending the minikube upgrade, let's try to synchronize things. Set a time to do the minikube upgrade this PM, if that works for you, and I'll merge that PR once it's done. Then I'll take care of backporting it to 2.4 and make sure it passes the integration tests. -- Marcelo >>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
i think we should. the version of k8s we were testing against before was v1.10.x, and we are currently testing master *and* 2.4.1 against v1.13.1. given that we're a couple of weeks away from v.1.14.x i feel that this is a safe and prudent thing to include. On Thu, Mar 14, 2019 at 10:57 AM DB Tsai wrote: > Since rc8 was already cut without the k8s client upgrade; the build is > ready to vote, and including k8s client upgrade in 2.4.1 implies that > we will drop the old-but-not-that-old > K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? > > Thanks. > > Sincerely, > > DB Tsai > -- > Web: https://www.dbtsai.com > PGP Key ID: 42E5B25A8F7A82C1 > > On Thu, Mar 14, 2019 at 9:48 AM shane knapp wrote: > > > > thanks everyone, both PRs are merged. :) > > > > On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > >> > >> btw, let's wait and see if the non-k8s PRB tests pass before merging > https://github.com/apache/spark/pull/23993 in to 2.4.1 > >> > >> On Wed, Mar 13, 2019 at 3:42 PM shane knapp > wrote: > >>> > >>> 2.4.1 k8s integration test passed: > >>> > >>> > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > >>> > >>> thanks everyone! :) > >>> > >>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp > wrote: > > 2.4.1 integration tests running: > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > > On Wed, Mar 13, 2019 at 3:15 PM shane knapp > wrote: > > > > upgrade completed, jenkins building again... master PR merged, > waiting for the 2.4.1 PR to launch the k8s integration tests. > > > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp > wrote: > >> > >> okie dokie! the time approacheth! > >> > >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect > the upgrade to take more than 15-20 mins, following which i will re-enable > builds. > >> > >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp > wrote: > >>> > >>> ok awesome. let's shoot for 3pm PST. > >>> > >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin < > van...@cloudera.com> wrote: > > On Wed, Mar 13, 2019 at 11:53 AM shane knapp > wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin < > van...@cloudera.com> wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master > client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the > minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge > that > PR once it's done. Then I'll take care of backporting it to 2.4 > and > make sure it passes the integration tests. > > -- > Marcelo > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
IMO we can vote on rc8; that upgrade should not block it. If it fails for some other reason, then rc9 will have it. Supported versions in k8s-land are really weird and don't match well to our release schedule. There isn't a good solution to this at the moment... good thing k8s support is still marked as experimental. :-) On Thu, Mar 14, 2019 at 10:56 AM DB Tsai wrote: > > Since rc8 was already cut without the k8s client upgrade; the build is > ready to vote, and including k8s client upgrade in 2.4.1 implies that > we will drop the old-but-not-that-old > K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? > > Thanks. > > Sincerely, > > DB Tsai > -- > Web: https://www.dbtsai.com > PGP Key ID: 42E5B25A8F7A82C1 > > On Thu, Mar 14, 2019 at 9:48 AM shane knapp wrote: > > > > thanks everyone, both PRs are merged. :) > > > > On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > >> > >> btw, let's wait and see if the non-k8s PRB tests pass before merging > >> https://github.com/apache/spark/pull/23993 in to 2.4.1 > >> > >> On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: > >>> > >>> 2.4.1 k8s integration test passed: > >>> > >>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > >>> > >>> thanks everyone! :) > >>> > >>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: > > 2.4.1 integration tests running: > https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ > > On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > > > upgrade completed, jenkins building again... master PR merged, waiting > > for the 2.4.1 PR to launch the k8s integration tests. > > > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > >> > >> okie dokie! the time approacheth! > >> > >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > >> upgrade to take more than 15-20 mins, following which i will re-enable > >> builds. > >> > >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp > >> wrote: > >>> > >>> ok awesome. let's shoot for 3pm PST. > >>> > >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin > >>> wrote: > > On Wed, Mar 13, 2019 at 11:53 AM shane knapp > wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > > wrote: > >> > >> Do the upgraded minikube/k8s versions break the current master > >> client > >> version too? > >> > > yes. > > Ah, so that part kinda sucks. > > Let's do this: since the master PR is good to go pending the minikube > upgrade, let's try to synchronize things. Set a time to do the > minikube upgrade this PM, if that works for you, and I'll merge that > PR once it's done. Then I'll take care of backporting it to 2.4 and > make sure it passes the integration tests. > > -- > Marcelo > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu > > > > > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu -- Marcelo - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
Since rc8 was already cut without the k8s client upgrade; the build is ready to vote, and including k8s client upgrade in 2.4.1 implies that we will drop the old-but-not-that-old K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Thu, Mar 14, 2019 at 9:48 AM shane knapp wrote: > > thanks everyone, both PRs are merged. :) > > On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: >> >> btw, let's wait and see if the non-k8s PRB tests pass before merging >> https://github.com/apache/spark/pull/23993 in to 2.4.1 >> >> On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: >>> >>> 2.4.1 k8s integration test passed: >>> >>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ >>> >>> thanks everyone! :) >>> >>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: 2.4.1 integration tests running: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: > > upgrade completed, jenkins building again... master PR merged, waiting > for the 2.4.1 PR to launch the k8s integration tests. > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: >> >> okie dokie! the time approacheth! >> >> i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the >> upgrade to take more than 15-20 mins, following which i will re-enable >> builds. >> >> On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: >>> >>> ok awesome. let's shoot for 3pm PST. >>> >>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin >>> wrote: On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: >> >> Do the upgraded minikube/k8s versions break the current master >> client >> version too? >> > yes. Ah, so that part kinda sucks. Let's do this: since the master PR is good to go pending the minikube upgrade, let's try to synchronize things. Set a time to do the minikube upgrade this PM, if that works for you, and I'll merge that PR once it's done. Then I'll take care of backporting it to 2.4 and make sure it passes the integration tests. -- Marcelo >>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu > > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update
thanks everyone, both PRs are merged. :) On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > btw, let's wait and see if the non-k8s PRB tests pass before merging > https://github.com/apache/spark/pull/23993 in to 2.4.1 > > On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: > >> 2.4.1 k8s integration test passed: >> >> >> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ >> >> thanks everyone! :) >> >> On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: >> >>> 2.4.1 integration tests running: >>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ >>> >>> On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote: >>> upgrade completed, jenkins building again... master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests. On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > okie dokie! the time approacheth! > > i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the > upgrade to take more than 15-20 mins, following which i will re-enable > builds. > > On Wed, Mar 13, 2019 at 12:17 PM shane knapp > wrote: > >> ok awesome. let's shoot for 3pm PST. >> >> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin >> wrote: >> >>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp >>> wrote: >>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin < >>> van...@cloudera.com> wrote: >>> >> >>> >> Do the upgraded minikube/k8s versions break the current master >>> client >>> >> version too? >>> >> >>> > yes. >>> >>> Ah, so that part kinda sucks. >>> >>> Let's do this: since the master PR is good to go pending the minikube >>> upgrade, let's try to synchronize things. Set a time to do the >>> minikube upgrade this PM, if that works for you, and I'll merge that >>> PR once it's done. Then I'll take care of backporting it to 2.4 and >>> make sure it passes the integration tests. >>> >>> -- >>> Marcelo >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
