Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

It's come to my attention that there have been several bug fixes merged since RC3: - SPARK-12404 - Fix serialization error for Datasets with Timestamps/Arrays/Decimal - SPARK-12218 - Fix incorrect pushdown of filters to parquet - SPARK-12395 - Fix join columns of outer join for DataFrame usi

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) All the test passed, and run it on HDP 2.3.2 sandbox successfully. On Sun, Dec 20, 2015 at 10:43 AM, Luciano Resende wrote: > +1 (non-binding) > > Tested Standalone mode, SparkR and couple Stream Apps, all seem ok. > > On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust > wrote:

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tested Standalone mode, SparkR and couple Stream Apps, all seem ok. On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December 19, 2015 at 18:00 UTC a

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Testing environment: -CDH5.5 single node docker -Prebuilt spark-1.6.0-hadoop2.6.tgz -Yarn-cluster mode Comparing outputs of Spark 1.5.x and 1.6.0-RC3: Pyspark OK?: K-Means (ml) - Note: our tests show a numerical diff here compared to the 1.5.2 output. Since K-Means has a random

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tests the without-hadoop binaries (so didn't run Hive-related tests) with a test batch including standalone / client, yarn / client and cluster, including core, mllib and streaming (flume and kafka). On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust wrote: > Please vote on relea

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tested a number of tests surrounding DataFrames, Datasets, and ML. On Wed, Dec 16, 2015 at 1:32 PM Michael Armbrust wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December 19, 2015 at 18:00 UTC an

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Thanks Sean for sending me the logs offline. Turns out the tests are failing again, for reasons unrelated to Spark. I have filed https://issues.apache.org/jira/browse/SPARK-12426 for that with some details. In the meanwhile, I agree with Sean, these tests should be disabled. And, again, I don't th

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Yes that's what I mean. If they're not quite working, let's disable them, but first, we have to rule out that I'm not just missing some requirement. Functionally, it's not worth blocking the release. It seems like bad form to release with tests that always fail for a non-trivial number of users, b

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Sean, Are you referring to docker integration tests? If so, they were disabled for majority of the release and I recently worked on it (SPARK-11796) and once it got committed, the tests were re-enabled in Spark builds. I am not sure what OSs the test builds use, but it should be passing there too.

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

For me, mostly the same as before: tests are mostly passing, but I can never get the docker tests to pass. If anyone knows a special profile or package that needs to be enabled, I can try that and/or fix/document it. Just wondering if it's me. I'm on Java 7 + Ubuntu 15.10, with -Pyarn -Phive -Phiv

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1.  Ran some regression tests on Spark on Yarn (hadoop 2.6 and 2.7). Tom On Wednesday, December 16, 2015 3:32 PM, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December 19, 2015 at 18:00 UTC and

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) It passes our tests after we registered 6 new classes with Kryo: kryo.register(classOf[org.apache.spark.sql.catalyst.expressions.UnsafeRow]) kryo.register(classOf[Array[org.apache.spark.mllib.tree.model.Split]]) kryo.register(Class.forName("org.apache.spark.mllib.tree.m

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 29:32 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib (iPython 4.0) 2.0 Spark version is 1.6.0 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

One correction, the better way is to just create a file called java-opts in .../spark/conf with the following config value in it -Dhdp.version=. One way to get the HDP version is to run the below one lines on a node of your HDP cluster. hdp-select status hadoop-client | sed 's/hadoop-client - \(

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Agree with Andrew, we shouldn't block the release for this. This issue won't be there in Spark distribution from Hortonworks since we set the HDP version. If you want to use the Apache Spark with HDP you can modify mapred-site.xml to replace the hdp.version property with the right value for your

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

The fix for the Mesos cluster regression has introduced another Mesos cluster bug. Namely, the MesosClusterDispatcher crashes when trying to write to ZK: https://issues.apache.org/jira/browse/SPARK-12413 I have a tentative fix here: https://github.com/apache/spark/pull/10366 On Thu, Dec 17, 2015

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

That seems like an HDP-specific issue. I did a quick search on "spark bad substitution" and all the results have to do with people failing to run YARN cluster in HDP. Here is a workaround

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

@Andrew Thanks for the reply, did you run this in a Hortonworks or Cloudera cluster? I suspect the issue is coming from the ​extraJavaOptions as these are necessary in HDP, the strange thing is that with exactly the same settings 1.5 works. # jar -tf spark-assembly-1.6.0-SNAPSHOT-hadoop2.7.1.jar |

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

@syepes I just run Spark 1.6 (881f254) on YARN with Hadoop 2.4.0. I was able to run a simple application in cluster mode successfully. Can you verify whether the org.apache.spark.yarn.ApplicationMaster class exists in your assembly jar? jar -tf assembly.jar | grep ApplicationMaster -Andrew 20

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

-1 (YARN Cluster deployment mode not working) I have just tested 1.6 (d509194b) on our HDP 2.3 platform and the cluster mode does not seem work. It looks like some parameter are not being passed correctly. This example works correctly with 1.5. # spark-submit --master yarn --deploy-mode cluster -

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Thursday, December 17, 2015 8:22 AM, Kousuke Saruta wrote: +1 On 2015/12/17 6:32, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December 19, 2015 at 18:00 UTC and passes if a m

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On 2015/12/17 6:32, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December 19, 2015 at 18:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

-0 (non-binding) Unfortunately the Mesos cluster regression is still there (see my comment for explanations). I'm not voting to delay the release any longer though. We tested (and passed) Mesos in: - client mode - fine/coarse-grained

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non binding) Tested in standalone and yarn with different samples. Regards JB On 12/16/2015 10:32 PM, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December 19, 2015 at 18:00 UTC and passes if a maj

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 7:19 PM, Patrick Wendell wrote: > +1 > > On Wed, Dec 16, 2015 at 6:15 PM, Ted Yu wrote: > >> Ran test suite (minus docker-integration-tests) >> All passed >> >> +1 >> >> [INFO] Spark Project External ZeroMQ .. SUCCESS [ >> 13.647 s] >> [INFO] Spark

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 6:15 PM, Ted Yu wrote: > Ran test suite (minus docker-integration-tests) > All passed > > +1 > > [INFO] Spark Project External ZeroMQ .. SUCCESS [ > 13.647 s] > [INFO] Spark Project External Kafka ... SUCCESS [ > 45.424 s] > [INF

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Ran test suite (minus docker-integration-tests) All passed +1 [INFO] Spark Project External ZeroMQ .. SUCCESS [ 13.647 s] [INFO] Spark Project External Kafka ... SUCCESS [ 45.424 s] [INFO] Spark Project Examples . SUCCESS [02:06

Re: Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) after SPARK-12345 is merged. On Thu, Dec 17, 2015 at 9:55 AM, Allen Zhang wrote: > plus 1 > > > > > > > 在 2015-12-17 09:39:39，"Joseph Bradley" 写道： > > +1 > > On Wed, Dec 16, 2015 at 5:26 PM, Reynold Xin wrote: > >> +1 >> >> >> On Wed, Dec 16, 2015 at 5:24 PM, Mark Hamstra >>

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 5:26 PM, Reynold Xin wrote: > +1 > > > On Wed, Dec 16, 2015 at 5:24 PM, Mark Hamstra > wrote: > >> +1 >> >> On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust > > wrote: >> >>> Please vote on releasing the following candidate as Apache Spark version >>> 1.6.0! >>> >>>

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 5:24 PM, Mark Hamstra wrote: > +1 > > On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust > wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 1.6.0! >> >> The vote is open until Saturday, December 19, 2015 at 18:00 UTC and >> passe

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December 19, 2015 at 18:00 UTC and > passes if a majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release t

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 4:37 PM, Andrew Or wrote: > +1 > > Mesos cluster mode regression in RC2 is now fixed (SPARK-12345 > / PR10332 > ). > > Also tested on standalone client and cluster mode. No

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 Mesos cluster mode regression in RC2 is now fixed (SPARK-12345 / PR10332 ). Also tested on standalone client and cluster mode. No problems. 2015-12-16 15:16 GMT-08:00 Rad Gruchalski : > I also not

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

I also noticed that spark.replClassServer.host and spark.replClassServer.port aren’t used anymore. The transport now happens over the main RpcEnv. Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/in/radgru

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

I was going to say that spark.executor.port is not used anymore in 1.6, but damn, there's still that akka backend hanging around there even when netty is being used... we should fix this, should be a simple one-liner. On Wed, Dec 16, 2015 at 2:35 PM, singinpirate wrote: > -0 (non-binding) > > I h

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

-0 (non-binding) I have observed that when we set spark.executor.port in 1.6, we get thrown a NPE in SparkEnv$.create(SparkEnv.scala:259). It used to work in 1.5.2. Is anyone else seeing this? On Wed, Dec 16, 2015 at 2:26 PM Jiří Syrový wrote: > +1 Tested in standalone mode and so far seems to

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 Tested in standalone mode and so far seems to be fairly stable. 2015-12-16 22:32 GMT+01:00 Michael Armbrust : > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December 19, 2015 at 18:00 UTC and > passes if a majority of at