Re: [VOTE] Release Apache Spark 2.4.2

2019-04-29 Thread Dongjoon Hyun
Hi, All and Xiao (as a next release manager). In any case, can the release manager include the information about the used release script as a part of VOTE email officially? That information will be very helpful to reproduce Spark build (in the downstream environment) Currently, it's not clearly

Re: [VOTE] Release Apache Spark 2.4.2

2019-04-29 Thread Wenchen Fan
> it could just be fixed in master rather than back-port and re-roll the RC I don't think the release script is part of the released product. That said, we can just fix the release script in branch 2.4 without creating a new RC. We can even create a new repo for the release script, like spark-web

Re: [VOTE] Release Apache Spark 2.4.2

2019-04-29 Thread Sean Owen
I think this is a reasonable idea; I know @vanzin had suggested it was simpler to use the latest in case a bug was found in the release script and then it could just be fixed in master rather than back-port and re-roll the RC. That said I think we did / had to already drop the ability to build <= 2

Re: Spark build can't find javac

2019-04-29 Thread Sean Owen
Your JAVA_HOME is pointing to a JRE rather than JDK installation. Or you've actually installed the JRE. Only the JDK has javac, etc. On Mon, Apr 29, 2019 at 4:36 PM Shmuel Blitz wrote: > Hi, > > Trying to build Spark on Manjaro with OpenJDK version 1.8.0_212, and I'm > getting the following erro

Spark build can't find javac

2019-04-29 Thread Shmuel Blitz
Hi, Trying to build Spark on Manjaro with OpenJDK version 1.8.0_212, and I'm getting the following error: Cannot run program "/usr/lib/jvm/java-8-openjdk/jre/bin/javac": error=2, No such file or directory > which javac /usr/bin/javac only when I set JAVA_HOME as follows, do I get it to run. > e

Re: DataSourceV2 sync, 17 April 2019

2019-04-29 Thread Ryan Blue
That is mostly correct. V2 standardizes the behavior of logical operations like CTAS across data sources, so it isn't compatible with v1 behavior. Consequently, we can't just move to v2 easily. We have to maintain both in parallel and eventually deprecate v1. We are aiming to have a working v2 in

Spark 2.4.1 on Kubernetes - DNS resolution of driver fails

2019-04-29 Thread Olivier Girardot
Hi everyone, I have ~300 spark job on Kubernetes (GKE) using the cluster auto-scaler, and sometimes while running these jobs a pretty bad thing happens, the driver (in cluster mode) gets scheduled on Kubernetes and launches many executor pods. So far so good, but the k8s "Service" associated to the