Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Hey All, The issue that Josh pointed out is not just a test failure, it's an issue with an important bug fix that was not correctly back-ported into the 1.4 branch. Unfortunately the overall state of the 1.4 branch tests on Jenkins was not in great shape so this was missed earlier on. Given that this is fixed now, I have prepared another RC and am leaning towards restarting the vote. If anyone feels strongly one way or the other let me know, otherwise I'll restart it in a few hours. I figured since this will likely finalize over the weekend anyways, it's not so bad to wait 1 additional day in order to get that fix. - Patrick On Wed, Jul 8, 2015 at 12:00 PM, Josh Rosen wrote: > I've filed https://issues.apache.org/jira/browse/SPARK-8903 to fix the > DataFrameStatSuite test failure. The problem turned out to be caused by a > mistake made while resolving a merge-conflict when backporting that patch to > branch-1.4. > > I've submitted https://github.com/apache/spark/pull/7295 to fix this issue. > > On Wed, Jul 8, 2015 at 11:30 AM, Sean Owen wrote: >> >> I see, but shouldn't this test not be run when Hive isn't in the build? >> >> On Wed, Jul 8, 2015 at 7:13 PM, Andrew Or wrote: >> > @Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and >> > `-Phive-thriftserver`. The MissingRequirementsError is just complaining >> > that >> > it can't find the right classes. The other one (DataFrameStatSuite) is a >> > little more concerning. >> > >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
I've filed https://issues.apache.org/jira/browse/SPARK-8903 to fix the DataFrameStatSuite test failure. The problem turned out to be caused by a mistake made while resolving a merge-conflict when backporting that patch to branch-1.4. I've submitted https://github.com/apache/spark/pull/7295 to fix this issue. On Wed, Jul 8, 2015 at 11:30 AM, Sean Owen wrote: > I see, but shouldn't this test not be run when Hive isn't in the build? > > On Wed, Jul 8, 2015 at 7:13 PM, Andrew Or wrote: > > @Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and > > `-Phive-thriftserver`. The MissingRequirementsError is just complaining > that > > it can't find the right classes. The other one (DataFrameStatSuite) is a > > little more concerning. > > > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
I see, but shouldn't this test not be run when Hive isn't in the build? On Wed, Jul 8, 2015 at 7:13 PM, Andrew Or wrote: > @Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and > `-Phive-thriftserver`. The MissingRequirementsError is just complaining that > it can't find the right classes. The other one (DataFrameStatSuite) is a > little more concerning. > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
@Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and `-Phive-thriftserver`. The MissingRequirementsError is just complaining that it can't find the right classes. The other one (DataFrameStatSuite) is a little more concerning. 2015-07-08 10:43 GMT-07:00 Pradeep Bashyal : > Hi Shivaram, > > I created a Jira Issue for the documentation error. > https://issues.apache.org/jira/browse/SPARK-8901 > > Thanks > Pradeep > > On Wed, Jul 8, 2015 at 11:40 AM, Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > >> Hi Pradeep >> >> Thanks for the catch -- Lets open a JIRA and PR for it. I don't think >> documentation changes affect the release though Patrick can confirm that. >> >> Thanks >> Shivaram >> >> On Wed, Jul 8, 2015 at 9:35 AM, Pradeep Bashyal >> wrote: >> >>> Here's one thing I ran into: >>> >>> The SparkR documentation example in >>> http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is >>> incorrect. >>> >>> sc <- sparkR.init(packages="com.databricks:spark-csv_2.11:1.0.3") >>> >>> should be >>> >>> sc <- sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3") >>> >>> >>> Thanks >>> Pradeep >>> >>> >>> On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen wrote: >>> The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark-1.4.1 > > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= > 3e8ae38944f13895daf328555c1ad22cd590b089 > > The release files, including signatures, digests, etc. can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/pwendell.asc > > The staging repository for this release can be found at: > [published as version: 1.4.1] > https://repository.apache.org/content/repositories/orgapachespark-1123/ > [published as version: 1.4.1-rc3] > https://repository.apache.org/content/repositories/orgapachespark-1124/ > > The documentation corresponding to this release can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ > > Please vote on releasing this package as Apache Spark 1.4.1! > > The vote is open until Friday, July 10, at 20:00 UTC and passes > if a majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Spark 1.4.1 > [ ] -1 Do not release this package because ... > > To learn more about Apache Spark, please see > http://spark.apache.org/ > > - > To unsubscribe, e-mail: dev-un
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Hi Shivaram, I created a Jira Issue for the documentation error. https://issues.apache.org/jira/browse/SPARK-8901 Thanks Pradeep On Wed, Jul 8, 2015 at 11:40 AM, Shivaram Venkataraman < shiva...@eecs.berkeley.edu> wrote: > Hi Pradeep > > Thanks for the catch -- Lets open a JIRA and PR for it. I don't think > documentation changes affect the release though Patrick can confirm that. > > Thanks > Shivaram > > On Wed, Jul 8, 2015 at 9:35 AM, Pradeep Bashyal > wrote: > >> Here's one thing I ran into: >> >> The SparkR documentation example in >> http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is >> incorrect. >> >> sc <- sparkR.init(packages="com.databricks:spark-csv_2.11:1.0.3") >> >> should be >> >> sc <- sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3") >> >> >> Thanks >> Pradeep >> >> >> On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen wrote: >> >>> The POM issue is resolved and the build succeeds. The license and sigs >>> still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the >>> following two exceptions. Is anyone else seeing these? this is >>> consistent on Ubuntu 14 with Java 7/8: >>> >>> DataFrameStatSuite: >>> ... >>> - special crosstab elements (., '', null, ``) *** FAILED *** >>> java.lang.NullPointerException: >>> at >>> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) >>> at >>> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) >>> at >>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>> at >>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>> at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) >>> at >>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >>> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >>> at >>> org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) >>> at >>> org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) >>> at >>> org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) >>> ... >>> >>> HiveSparkSubmitSuite: >>> - SPARK-8368: includes jars passed in through --jars *** FAILED *** >>> Process returned with exit code 1. See the log4j logs for more >>> detail. (HiveSparkSubmitSuite.scala:92) >>> - SPARK-8020: set sql conf in spark conf *** FAILED *** >>> Process returned with exit code 1. See the log4j logs for more >>> detail. (HiveSparkSubmitSuite.scala:92) >>> - SPARK-8489: MissingRequirementError during reflection *** FAILED *** >>> Process returned with exit code 1. See the log4j logs for more >>> detail. (HiveSparkSubmitSuite.scala:92) >>> >>> On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell >>> wrote: >>> > Please vote on releasing the following candidate as Apache Spark >>> version 1.4.1! >>> > >>> > This release fixes a handful of known issues in Spark 1.4.0, listed >>> here: >>> > http://s.apache.org/spark-1.4.1 >>> > >>> > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): >>> > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= >>> > 3e8ae38944f13895daf328555c1ad22cd590b089 >>> > >>> > The release files, including signatures, digests, etc. can be found at: >>> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ >>> > >>> > Release artifacts are signed with the following key: >>> > https://people.apache.org/keys/committer/pwendell.asc >>> > >>> > The staging repository for this release can be found at: >>> > [published as version: 1.4.1] >>> > >>> https://repository.apache.org/content/repositories/orgapachespark-1123/ >>> > [published as version: 1.4.1-rc3] >>> > >>> https://repository.apache.org/content/repositories/orgapachespark-1124/ >>> > >>> > The documentation corresponding to this release can be found at: >>> > >>> http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ >>> > >>> > Please vote on releasing this package as Apache Spark 1.4.1! >>> > >>> > The vote is open until Friday, July 10, at 20:00 UTC and passes >>> > if a majority of at least 3 +1 PMC votes are cast. >>> > >>> > [ ] +1 Release this package as Apache Spark 1.4.1 >>> > [ ] -1 Do not release this package because ... >>> > >>> > To learn more about Apache Spark, please see >>> > http://spark.apache.org/ >>> > >>> > - >>> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>> > For additional commands, e-mail: dev-h...@spark.apache.org >>> > >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>> For additional commands, e-mail: dev-h...@spark.apache.org >>> >>> >> >
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Yeah - we can fix the docs separately from the release. - Patrick On Wed, Jul 8, 2015 at 10:03 AM, Mark Hamstra wrote: > HiveSparkSubmitSuite is fine for me, but I do see the same issue with > DataFrameStatSuite -- OSX 10.10.4, java > > 1.7.0_75, -Phive -Phive-thriftserver -Phadoop-2.4 -Pyarn > > > On Wed, Jul 8, 2015 at 4:18 AM, Sean Owen wrote: >> >> The POM issue is resolved and the build succeeds. The license and sigs >> still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the >> following two exceptions. Is anyone else seeing these? this is >> consistent on Ubuntu 14 with Java 7/8: >> >> DataFrameStatSuite: >> ... >> - special crosstab elements (., '', null, ``) *** FAILED *** >> java.lang.NullPointerException: >> at >> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) >> at >> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) >> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >> at >> org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) >> at >> org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) >> at >> org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) >> ... >> >> HiveSparkSubmitSuite: >> - SPARK-8368: includes jars passed in through --jars *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> - SPARK-8020: set sql conf in spark conf *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> - SPARK-8489: MissingRequirementError during reflection *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> >> On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell >> wrote: >> > Please vote on releasing the following candidate as Apache Spark version >> > 1.4.1! >> > >> > This release fixes a handful of known issues in Spark 1.4.0, listed >> > here: >> > http://s.apache.org/spark-1.4.1 >> > >> > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): >> > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= >> > 3e8ae38944f13895daf328555c1ad22cd590b089 >> > >> > The release files, including signatures, digests, etc. can be found at: >> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ >> > >> > Release artifacts are signed with the following key: >> > https://people.apache.org/keys/committer/pwendell.asc >> > >> > The staging repository for this release can be found at: >> > [published as version: 1.4.1] >> > https://repository.apache.org/content/repositories/orgapachespark-1123/ >> > [published as version: 1.4.1-rc3] >> > https://repository.apache.org/content/repositories/orgapachespark-1124/ >> > >> > The documentation corresponding to this release can be found at: >> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ >> > >> > Please vote on releasing this package as Apache Spark 1.4.1! >> > >> > The vote is open until Friday, July 10, at 20:00 UTC and passes >> > if a majority of at least 3 +1 PMC votes are cast. >> > >> > [ ] +1 Release this package as Apache Spark 1.4.1 >> > [ ] -1 Do not release this package because ... >> > >> > To learn more about Apache Spark, please see >> > http://spark.apache.org/ >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> > For additional commands, e-mail: dev-h...@spark.apache.org >> > >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
HiveSparkSubmitSuite is fine for me, but I do see the same issue with DataFrameStatSuite -- OSX 10.10.4, java 1.7.0_75, -Phive -Phive-thriftserver -Phadoop-2.4 -Pyarn On Wed, Jul 8, 2015 at 4:18 AM, Sean Owen wrote: > The POM issue is resolved and the build succeeds. The license and sigs > still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the > following two exceptions. Is anyone else seeing these? this is > consistent on Ubuntu 14 with Java 7/8: > > DataFrameStatSuite: > ... > - special crosstab elements (., '', null, ``) *** FAILED *** > java.lang.NullPointerException: > at > org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) > at > org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at > org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) > at > org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) > at > org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) > ... > > HiveSparkSubmitSuite: > - SPARK-8368: includes jars passed in through --jars *** FAILED *** > Process returned with exit code 1. See the log4j logs for more > detail. (HiveSparkSubmitSuite.scala:92) > - SPARK-8020: set sql conf in spark conf *** FAILED *** > Process returned with exit code 1. See the log4j logs for more > detail. (HiveSparkSubmitSuite.scala:92) > - SPARK-8489: MissingRequirementError during reflection *** FAILED *** > Process returned with exit code 1. See the log4j logs for more > detail. (HiveSparkSubmitSuite.scala:92) > > On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell > wrote: > > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > > http://s.apache.org/spark-1.4.1 > > > > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= > > 3e8ae38944f13895daf328555c1ad22cd590b089 > > > > The release files, including signatures, digests, etc. can be found at: > > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/pwendell.asc > > > > The staging repository for this release can be found at: > > [published as version: 1.4.1] > > https://repository.apache.org/content/repositories/orgapachespark-1123/ > > [published as version: 1.4.1-rc3] > > https://repository.apache.org/content/repositories/orgapachespark-1124/ > > > > The documentation corresponding to this release can be found at: > > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ > > > > Please vote on releasing this package as Apache Spark 1.4.1! > > > > The vote is open until Friday, July 10, at 20:00 UTC and passes > > if a majority of at least 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Spark 1.4.1 > > [ ] -1 Do not release this package because ... > > > > To learn more about Apache Spark, please see > > http://spark.apache.org/ > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > For additional commands, e-mail: dev-h...@spark.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Hi Pradeep Thanks for the catch -- Lets open a JIRA and PR for it. I don't think documentation changes affect the release though Patrick can confirm that. Thanks Shivaram On Wed, Jul 8, 2015 at 9:35 AM, Pradeep Bashyal wrote: > Here's one thing I ran into: > > The SparkR documentation example in > http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is > incorrect. > > sc <- sparkR.init(packages="com.databricks:spark-csv_2.11:1.0.3") > > should be > > sc <- sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3") > > > Thanks > Pradeep > > > On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen wrote: > >> The POM issue is resolved and the build succeeds. The license and sigs >> still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the >> following two exceptions. Is anyone else seeing these? this is >> consistent on Ubuntu 14 with Java 7/8: >> >> DataFrameStatSuite: >> ... >> - special crosstab elements (., '', null, ``) *** FAILED *** >> java.lang.NullPointerException: >> at >> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) >> at >> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) >> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >> at >> org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) >> at >> org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) >> at >> org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) >> ... >> >> HiveSparkSubmitSuite: >> - SPARK-8368: includes jars passed in through --jars *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> - SPARK-8020: set sql conf in spark conf *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> - SPARK-8489: MissingRequirementError during reflection *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> >> On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell >> wrote: >> > Please vote on releasing the following candidate as Apache Spark >> version 1.4.1! >> > >> > This release fixes a handful of known issues in Spark 1.4.0, listed >> here: >> > http://s.apache.org/spark-1.4.1 >> > >> > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): >> > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= >> > 3e8ae38944f13895daf328555c1ad22cd590b089 >> > >> > The release files, including signatures, digests, etc. can be found at: >> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ >> > >> > Release artifacts are signed with the following key: >> > https://people.apache.org/keys/committer/pwendell.asc >> > >> > The staging repository for this release can be found at: >> > [published as version: 1.4.1] >> > https://repository.apache.org/content/repositories/orgapachespark-1123/ >> > [published as version: 1.4.1-rc3] >> > https://repository.apache.org/content/repositories/orgapachespark-1124/ >> > >> > The documentation corresponding to this release can be found at: >> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ >> > >> > Please vote on releasing this package as Apache Spark 1.4.1! >> > >> > The vote is open until Friday, July 10, at 20:00 UTC and passes >> > if a majority of at least 3 +1 PMC votes are cast. >> > >> > [ ] +1 Release this package as Apache Spark 1.4.1 >> > [ ] -1 Do not release this package because ... >> > >> > To learn more about Apache Spark, please see >> > http://spark.apache.org/ >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> > For additional commands, e-mail: dev-h...@spark.apache.org >> > >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Although that should be fixed if it's incorrect, it's not something that would nearly block a release. The question here is whether this artifact can be released as 1.4.1, or whether it has a blocking regression from 1.4.0. On Wed, Jul 8, 2015 at 5:35 PM, Pradeep Bashyal wrote: > Here's one thing I ran into: > > The SparkR documentation example in > http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is > incorrect. > > sc <- sparkR.init(packages="com.databricks:spark-csv_2.11:1.0.3") > > should be > > sc <- sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3") > > > Thanks > Pradeep > > > On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen wrote: >> >> The POM issue is resolved and the build succeeds. The license and sigs >> still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the >> following two exceptions. Is anyone else seeing these? this is >> consistent on Ubuntu 14 with Java 7/8: >> >> DataFrameStatSuite: >> ... >> - special crosstab elements (., '', null, ``) *** FAILED *** >> java.lang.NullPointerException: >> at >> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) >> at >> org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) >> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >> at >> org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) >> at >> org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) >> at >> org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) >> ... >> >> HiveSparkSubmitSuite: >> - SPARK-8368: includes jars passed in through --jars *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> - SPARK-8020: set sql conf in spark conf *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> - SPARK-8489: MissingRequirementError during reflection *** FAILED *** >> Process returned with exit code 1. See the log4j logs for more >> detail. (HiveSparkSubmitSuite.scala:92) >> >> On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell >> wrote: >> > Please vote on releasing the following candidate as Apache Spark version >> > 1.4.1! >> > >> > This release fixes a handful of known issues in Spark 1.4.0, listed >> > here: >> > http://s.apache.org/spark-1.4.1 >> > >> > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): >> > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= >> > 3e8ae38944f13895daf328555c1ad22cd590b089 >> > >> > The release files, including signatures, digests, etc. can be found at: >> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ >> > >> > Release artifacts are signed with the following key: >> > https://people.apache.org/keys/committer/pwendell.asc >> > >> > The staging repository for this release can be found at: >> > [published as version: 1.4.1] >> > https://repository.apache.org/content/repositories/orgapachespark-1123/ >> > [published as version: 1.4.1-rc3] >> > https://repository.apache.org/content/repositories/orgapachespark-1124/ >> > >> > The documentation corresponding to this release can be found at: >> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ >> > >> > Please vote on releasing this package as Apache Spark 1.4.1! >> > >> > The vote is open until Friday, July 10, at 20:00 UTC and passes >> > if a majority of at least 3 +1 PMC votes are cast. >> > >> > [ ] +1 Release this package as Apache Spark 1.4.1 >> > [ ] -1 Do not release this package because ... >> > >> > To learn more about Apache Spark, please see >> > http://spark.apache.org/ >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> > For additional commands, e-mail: dev-h...@spark.apache.org >> > >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Here's one thing I ran into: The SparkR documentation example in http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is incorrect. sc <- sparkR.init(packages="com.databricks:spark-csv_2.11:1.0.3") should be sc <- sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3") Thanks Pradeep On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen wrote: > The POM issue is resolved and the build succeeds. The license and sigs > still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the > following two exceptions. Is anyone else seeing these? this is > consistent on Ubuntu 14 with Java 7/8: > > DataFrameStatSuite: > ... > - special crosstab elements (., '', null, ``) *** FAILED *** > java.lang.NullPointerException: > at > org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) > at > org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at > org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) > at > org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) > at > org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) > ... > > HiveSparkSubmitSuite: > - SPARK-8368: includes jars passed in through --jars *** FAILED *** > Process returned with exit code 1. See the log4j logs for more > detail. (HiveSparkSubmitSuite.scala:92) > - SPARK-8020: set sql conf in spark conf *** FAILED *** > Process returned with exit code 1. See the log4j logs for more > detail. (HiveSparkSubmitSuite.scala:92) > - SPARK-8489: MissingRequirementError during reflection *** FAILED *** > Process returned with exit code 1. See the log4j logs for more > detail. (HiveSparkSubmitSuite.scala:92) > > On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell > wrote: > > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > > http://s.apache.org/spark-1.4.1 > > > > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= > > 3e8ae38944f13895daf328555c1ad22cd590b089 > > > > The release files, including signatures, digests, etc. can be found at: > > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/pwendell.asc > > > > The staging repository for this release can be found at: > > [published as version: 1.4.1] > > https://repository.apache.org/content/repositories/orgapachespark-1123/ > > [published as version: 1.4.1-rc3] > > https://repository.apache.org/content/repositories/orgapachespark-1124/ > > > > The documentation corresponding to this release can be found at: > > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ > > > > Please vote on releasing this package as Apache Spark 1.4.1! > > > > The vote is open until Friday, July 10, at 20:00 UTC and passes > > if a majority of at least 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Spark 1.4.1 > > [ ] -1 Do not release this package because ... > > > > To learn more about Apache Spark, please see > > http://spark.apache.org/ > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > For additional commands, e-mail: dev-h...@spark.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark-1.4.1 > > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= > 3e8ae38944f13895daf328555c1ad22cd590b089 > > The release files, including signatures, digests, etc. can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/pwendell.asc > > The staging repository for this release can be found at: > [published as version: 1.4.1] > https://repository.apache.org/content/repositories/orgapachespark-1123/ > [published as version: 1.4.1-rc3] > https://repository.apache.org/content/repositories/orgapachespark-1124/ > > The documentation corresponding to this release can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ > > Please vote on releasing this package as Apache Spark 1.4.1! > > The vote is open until Friday, July 10, at 20:00 UTC and passes > if a majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Spark 1.4.1 > [ ] -1 Do not release this package because ... > > To learn more about Apache Spark, please see > http://spark.apache.org/ > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
+1 (non-binding, of course)
1. Compiled OSX 10.10 (Yosemite) OK Total time: 27:24 min
mvn clean package -Pyarn -Phadoop-2.6 -DskipTests
2. Tested pyspark, mllib
2.1. statistics (min,max,mean,Pearson,Spearman) OK
2.2. Linear/Ridge/Laso Regression OK
2.3. Decision Tree, Naive Bayes OK
2.4. KMeans OK
Center And Scale OK
2.5. RDD operations OK
State of the Union Texts - MapReduce, Filter,sortByKey (word count)
2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK
Model evaluation/optimization (rank, numIter, lambda) with itertools
OK
3. Scala - MLlib
3.1. statistics (min,max,mean,Pearson,Spearman) OK
3.2. LinearRegressionWithSGD OK
3.3. Decision Tree OK
3.4. KMeans OK
3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK
3.6. saveAsParquetFile OK
3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile,
registerTempTable, sql OK
3.8. result = sqlContext.sql("SELECT
OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER
JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID") OK
4.0. Spark SQL from Python OK
4.1. result = sqlContext.sql("SELECT * from people WHERE State = 'WA'") OK
5.0. Packages
5.1. com.databricks.spark.csv - read/write OK
6.0. DataFrames
6.1. cast,dtypes OK
6.2. groupBy,avg,crosstab,corr,isNull,na.drop OK
6.3. joins,sql,set operations,udf OK
Cheers
On Tue, Jul 7, 2015 at 12:06 PM, Patrick Wendell wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.4.1!
>
> This release fixes a handful of known issues in Spark 1.4.0, listed here:
> http://s.apache.org/spark-1.4.1
>
> The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38):
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=
> 3e8ae38944f13895daf328555c1ad22cd590b089
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> [published as version: 1.4.1]
> https://repository.apache.org/content/repositories/orgapachespark-1123/
> [published as version: 1.4.1-rc3]
> https://repository.apache.org/content/repositories/orgapachespark-1124/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/
>
> Please vote on releasing this package as Apache Spark 1.4.1!
>
> The vote is open until Friday, July 10, at 20:00 UTC and passes
> if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.4.1
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see
> http://spark.apache.org/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
+1 Verified that the previous blockers SPARK-8781 and SPARK-8819 are now resolved. 2015-07-07 12:06 GMT-07:00 Patrick Wendell : > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark-1.4.1 > > The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= > 3e8ae38944f13895daf328555c1ad22cd590b089 > > The release files, including signatures, digests, etc. can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/pwendell.asc > > The staging repository for this release can be found at: > [published as version: 1.4.1] > https://repository.apache.org/content/repositories/orgapachespark-1123/ > [published as version: 1.4.1-rc3] > https://repository.apache.org/content/repositories/orgapachespark-1124/ > > The documentation corresponding to this release can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ > > Please vote on releasing this package as Apache Spark 1.4.1! > > The vote is open until Friday, July 10, at 20:00 UTC and passes > if a majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Spark 1.4.1 > [ ] -1 Do not release this package because ... > > To learn more about Apache Spark, please see > http://spark.apache.org/ > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
