Re: [SS] Why does StreamingQueryManager.notifyQueryTermination use id and runId (not just id)?

stateStoreCoordinator uses runId to deal with a small chance that Spark cannot turn a bad task down. Please see https://github.com/apache/spark/pull/18355 On Fri, Oct 27, 2017 at 3:40 AM, Jacek Laskowski wrote: > Hi, > > I'm wondering why

Re: Anyone knows how to build and spark on jdk9?

Certainly, Scala 2.12 support precedes Java 9 support. A lot of the work is in place already, and the last issue is dealing with how Scala closures are now implemented quite different with lambdas / invokedynamic. This affects the ClosureCleaner. For the interested, this is as far as I know the

Re: Anyone knows how to build and spark on jdk9?

Scala 2.12 is not yet supported on Spark - this means also not JDK9: https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-14220 If you look at the Oracle support then jdk 9 is anyway only supported for 6 months. JDK 8 is Lts (5 years) JDK 18.3 will be only 6 months and JDK 18.9 is

Re: Anyone knows how to build and spark on jdk9?

If someone else is looking how to try jdk9, you can just pass your own JAVA_HOME environment variables: spark.yarn.appMasterEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0 spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-1.8.0 On Fri, Oct 27, 2017 at 5:14 AM, Steve Loughran wrote:

[SS] Why does StreamingQueryManager.notifyQueryTermination use id and runId (not just id)?

Hi, I'm wondering why StreamingQueryManager.notifyQueryTermination [1] use a query id to remove it from the activeQueries internal registry [2] while notifies stateStoreCoordinator using runId [3]? My understanding is that id is the same across different runs of a query so once

Re: Anyone knows how to build and spark on jdk9?

On 27 Oct 2017, at 03:21, Zhang, Liyun > wrote: Hi all: 1. I want to build spark on jdk9 and test it with Hadoop on jdk9 env. I search for jiras related to JDK9. I only found