alright, my canary build w/skipping the PV integration test passed w/the docker driver: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s-clone/20/
i'll put together a PR for this over the weekend (it's a one-liner) and once we merge i can get the remaining workers upgraded early next week. On Thu, Apr 15, 2021 at 3:05 PM shane knapp ☠ <skn...@berkeley.edu> wrote: > i'm all for that... and once they're turned off, we can finish the > minikube/k8s/move-to-docker project in a couple of hours max. > > On Thu, Apr 15, 2021 at 3:00 PM Holden Karau <hol...@pigscanfly.ca> wrote: > >> What about if we just turn off the PV tests for now? >> I'd be happy to help with the debugging/upgrading. >> >> On Thu, Apr 15, 2021 at 2:28 AM Rob Vesse <rve...@dotnetrdf.org> wrote: >> > >> > There’s at least one test (the persistent volumes one) that relies on >> some Minikube functionality because we run integration tests for our >> $dayjob Spark image builds using Docker for Desktop instead and that one >> test fails because it relies on some minikube specific functionality. That >> test could be refactored because I think it’s just adding a minimal Ceph >> cluster to the K8S cluster which can be done to any K8S cluster in principal >> > >> > >> > >> > Rob >> > >> > >> > >> > From: shane knapp ☠ <skn...@berkeley.edu> >> > Date: Wednesday, 14 April 2021 at 18:56 >> > To: Frank Luo <luohui...@gmail.com> >> > Cc: dev <dev@spark.apache.org>, Brian K Shiratsuki <b...@berkeley.edu> >> > Subject: Re: [SPARK-34738] issues w/k8s+minikube and PV tests >> > >> > >> > >> > On Wed, Apr 14, 2021 at 10:32 AM Frank Luo <luohui...@gmail.com> wrote: >> > >> > Is there any hard dependency on minkube? (i.e, GPU setting), kind ( >> https://kind.sigs.k8s.io/) is a stabler and simpler k8s cluster env on a >> single machine (only requires docker) , it been widely used by k8s projects >> testing. >> > >> > >> > >> > there are no hard deps on minikube... it installs happily and >> successfully runs every integration test except for persistent volumes. >> > >> > >> > >> > i haven't tried kind yet, but my time is super limited on this and i'd >> rather not venture down another rabbit hole unless we absolutely have to. >> > >> > >> >> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > Shane Knapp > Computer Guy / Voice of Reason > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp Computer Guy / Voice of Reason UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
