Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

It's come to my attention that there have been several bug fixes merged since RC3: - SPARK-12404 - Fix serialization error for Datasets with Timestamps/Arrays/Decimal - SPARK-12218 - Fix incorrect pushdown of filters to parquet - SPARK-12395 - Fix join columns of outer join for DataFrame

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tested Standalone mode, SparkR and couple Stream Apps, all seem ok. On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday,

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Testing environment: -CDH5.5 single node docker -Prebuilt spark-1.6.0-hadoop2.6.tgz -Yarn-cluster mode Comparing outputs of Spark 1.5.x and 1.6.0-RC3: Pyspark OK?: K-Means (ml) - Note: our tests show a numerical diff here compared to the 1.5.2 output. Since K-Means has a random

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) All the test passed, and run it on HDP 2.3.2 sandbox successfully. On Sun, Dec 20, 2015 at 10:43 AM, Luciano Resende wrote: > +1 (non-binding) > > Tested Standalone mode, SparkR and couple Stream Apps, all seem ok. > > On Wed, Dec 16, 2015 at 1:32 PM,

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) It passes our tests after we registered 6 new classes with Kryo: kryo.register(classOf[org.apache.spark.sql.catalyst.expressions.UnsafeRow]) kryo.register(classOf[Array[org.apache.spark.mllib.tree.model.Split]])

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

For me, mostly the same as before: tests are mostly passing, but I can never get the docker tests to pass. If anyone knows a special profile or package that needs to be enabled, I can try that and/or fix/document it. Just wondering if it's me. I'm on Java 7 + Ubuntu 15.10, with -Pyarn -Phive

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1.  Ran some regression tests on Spark on Yarn (hadoop 2.6 and 2.7). Tom On Wednesday, December 16, 2015 3:32 PM, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Yes that's what I mean. If they're not quite working, let's disable them, but first, we have to rule out that I'm not just missing some requirement. Functionally, it's not worth blocking the release. It seems like bad form to release with tests that always fail for a non-trivial number of users,

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Thanks Sean for sending me the logs offline. Turns out the tests are failing again, for reasons unrelated to Spark. I have filed https://issues.apache.org/jira/browse/SPARK-12426 for that with some details. In the meanwhile, I agree with Sean, these tests should be disabled. And, again, I don't

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tests the without-hadoop binaries (so didn't run Hive-related tests) with a test batch including standalone / client, yarn / client and cluster, including core, mllib and streaming (flume and kafka). On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding) Tested a number of tests surrounding DataFrames, Datasets, and ML. On Wed, Dec 16, 2015 at 1:32 PM Michael Armbrust wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 29:32 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib (iPython 4.0) 2.0 Spark version is 1.6.0 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Agree with Andrew, we shouldn't block the release for this. This issue won't be there in Spark distribution from Hortonworks since we set the HDP version. If you want to use the Apache Spark with HDP you can modify mapred-site.xml to replace the hdp.version property with the right value for

[VOTE] Release Apache Spark 1.6.0 (RC3)

Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December 19, 2015 at 18:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.6.0 [ ] -1 Do not release this package

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 Tested in standalone mode and so far seems to be fairly stable. 2015-12-16 22:32 GMT+01:00 Michael Armbrust : > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December 19, 2015 at 18:00 UTC and >

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

-0 (non-binding) I have observed that when we set spark.executor.port in 1.6, we get thrown a NPE in SparkEnv$.create(SparkEnv.scala:259). It used to work in 1.5.2. Is anyone else seeing this? On Wed, Dec 16, 2015 at 2:26 PM Jiří Syrový wrote: > +1 Tested in standalone

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

I was going to say that spark.executor.port is not used anymore in 1.6, but damn, there's still that akka backend hanging around there even when netty is being used... we should fix this, should be a simple one-liner. On Wed, Dec 16, 2015 at 2:35 PM, singinpirate

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

I also noticed that spark.replClassServer.host and spark.replClassServer.port aren’t used anymore. The transport now happens over the main RpcEnv. Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com)

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 (non binding) Tested in standalone and yarn with different samples. Regards JB On 12/16/2015 10:32 PM, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December 19, 2015 at 18:00 UTC and passes if a

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 7:19 PM, Patrick Wendell wrote: > +1 > > On Wed, Dec 16, 2015 at 6:15 PM, Ted Yu wrote: > >> Ran test suite (minus docker-integration-tests) >> All passed >> >> +1 >> >> [INFO] Spark Project External ZeroMQ

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 4:37 PM, Andrew Or wrote: > +1 > > Mesos cluster mode regression in RC2 is now fixed (SPARK-12345 > / PR10332 > ). > > Also tested on standalone

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Ran test suite (minus docker-integration-tests) All passed +1 [INFO] Spark Project External ZeroMQ .. SUCCESS [ 13.647 s] [INFO] Spark Project External Kafka ... SUCCESS [ 45.424 s] [INFO] Spark Project Examples . SUCCESS [02:06

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Saturday, December 19, 2015 at 18:00 UTC and > passes if a majority of at least 3 +1 PMC votes are

Re:Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

plus 1 在 2015-12-17 09:39:39，"Joseph Bradley" 写道： +1 On Wed, Dec 16, 2015 at 5:26 PM, Reynold Xin wrote: +1 On Wed, Dec 16, 2015 at 5:24 PM, Mark Hamstra wrote: +1 On Wed, Dec 16, 2015 at 1:32 PM, Michael

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 5:26 PM, Reynold Xin wrote: > +1 > > > On Wed, Dec 16, 2015 at 5:24 PM, Mark Hamstra > wrote: > >> +1 >> >> On Wed, Dec 16, 2015 at 1:32 PM, Michael Armbrust > > wrote: >> >>> Please vote on

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 On Wed, Dec 16, 2015 at 6:15 PM, Ted Yu wrote: > Ran test suite (minus docker-integration-tests) > All passed > > +1 > > [INFO] Spark Project External ZeroMQ .. SUCCESS [ > 13.647 s] > [INFO] Spark Project External Kafka ...