Re: [VOTE] Decommissioning SPIP

2020-07-06 Thread Terry Kim
+1 (non-binding) Thanks, Terry On Wed, Jul 1, 2020 at 6:05 PM Holden Karau wrote: > Hi Spark Devs, > > I think discussion has settled on the SPIP doc at > https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing > , > design doc at >

Re: m2 cache issues in Jenkins?

2020-07-06 Thread shane knapp ☠
once the jobs running on that worker are finished, yes. On Sun, Jul 5, 2020 at 7:41 PM Hyukjin Kwon wrote: > Shane, can we remove .m2 in worker machine 4? > > 2020년 7월 3일 (금) 오전 8:18, Jungtaek Lim 님이 작성: > >> Looks like Jenkins service itself becomes unstable. It took considerable >> time to

Re: [VOTE] Decommissioning SPIP

2020-07-06 Thread Tom Graves
+1 Tom On Wednesday, July 1, 2020, 08:05:47 PM CDT, Holden Karau wrote: Hi Spark Devs, I think discussion has settled on the SPIP doc at  https://docs.google.com/document/d/1EOei24ZpVvR7_w0BwBjOnrWRy4k-qTdIlx60FsHZSHA/edit?usp=sharing  , design doc at 

Re: m2 cache issues in Jenkins?

2020-07-06 Thread shane knapp ☠
i killed and retriggered the PRB jobs on 04, and wiped that workers' m2 cache. On Mon, Jul 6, 2020 at 9:24 AM shane knapp ☠ wrote: > once the jobs running on that worker are finished, yes. > > On Sun, Jul 5, 2020 at 7:41 PM Hyukjin Kwon wrote: > >> Shane, can we remove .m2 in worker machine 4?

Re: m2 cache issues in Jenkins?

2020-07-06 Thread shane knapp ☠
ok, i'm gonna have to reboot all the workers tomorrow and wipe the m2 caches. it looks like zombie builds were lingering post-jenkins-wedging and corrupting the repos. fixed on -05. On Mon, Jul 6, 2020 at 2:17 PM Jungtaek Lim wrote: > Just encountered the same and it's worker-05 again. (You

Re: [VOTE] Decommissioning SPIP

2020-07-06 Thread Holden Karau
Thank you to everyone who took the time to vote on this SPIP, review any of the PRs or design docs. The vote is now closed and the vote passes. I’m looking forward to trying to get a good base of this feature in for Spark 3.1. I’ll update the JIRA sub-components this afternoon and return to

Re: m2 cache issues in Jenkins?

2020-07-06 Thread Jungtaek Lim
Could this be a flaky or persistent issue? It failed with Scala gendoc but it didn't fail with the part the PR modified. It ran from worker-05. https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125121/consoleFull On Tue, Jul 7, 2020 at 2:10 AM shane knapp ☠ wrote: > i killed

Re: java.lang.ClassNotFoundException for s3a comitter

2020-07-06 Thread Stephen Coy
Hi Steve, While I understand your point regarding the mixing of Hadoop jars, this does not address the java.lang.ClassNotFoundException. Prebuilt Apache Spark 3.0 builds are only available for Hadoop 2.7 or Hadoop 3.2. Not Hadoop 3.1. The only place that I have found that missing class is in

Re: Spark 3 pod template for the driver

2020-07-06 Thread Edward Mitchell
If I had to guess, it's likely because the Spark code would have to read the YAML to make sure the required parameters are set, and the way it's done was just easier to build on without a lot of refactoring. On Mon, Jul 6, 2020 at 5:06 PM Michel Sumbul wrote: > Thanks Edward for the reply! > >

Re: m2 cache issues in Jenkins?

2020-07-06 Thread Jungtaek Lim
Just encountered the same and it's worker-05 again. (You can find [error] in the console to see what's the problem. I guess jetty artifacts in the worker might be messed up.) https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125127/consoleFull On Tue, Jul 7, 2020 at 5:35 AM