[jira] [Updated] (SPARK-5659) Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5659: --- Component/s: Tests > Flaky Test: org.apache.spark.streaming.ReceiverSuite.bl

[jira] [Commented] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311490#comment-14311490 ] Patrick Wendell commented on SPARK-761: --- [~aash] right now we don't e

[jira] [Updated] (SPARK-4687) SparkContext#addFile doesn't keep file folder information

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4687: --- Component/s: Spark Core > SparkContext#addFile doesn't keep file folder inf

[jira] [Updated] (SPARK-5299) Is http://www.apache.org/dist/spark/KEYS out of date?

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5299: --- Component/s: (was: Deploy) Build > Is http://www.apache.org/dist/sp

[jira] [Updated] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-761: -- Description: As a starter task, it would be good to audit the current behavior for different

[jira] [Updated] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-761: -- Labels: starter (was: ) > Print a nicer error message when incompatible Spark binaries try

[jira] [Commented] (SPARK-761) Print a nicer error message when incompatible Spark binaries try to talk

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311475#comment-14311475 ] Patrick Wendell commented on SPARK-761: --- I think the main thing to catch woul

[jira] [Updated] (SPARK-3033) [Hive] java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal

2015-02-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3033: --- Component/s: (was: Spark Core) > [Hive] java.math.BigDecimal cannot be cast

Unit tests

2015-02-08 Thread Patrick Wendell
Hey All, The tests are in a not-amazing state right now due to a few compounding factors: 1. We've merged a large volume of patches recently. 2. The load on jenkins has been relatively high, exposing races and other behavior not seen at lower load. For those not familiar, the main issue is flaky

Re: Improving metadata in Spark JIRA

2015-02-08 Thread Patrick Wendell
the way, isn't it possible to make the "Component" field mandatory when > people open new issues? Shouldn't we do that? > > Btw Patrick, don't we need a YARN component? I think our JIRA components > should roughly match the components on the PR dashboard. > >

[RESULT] [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-08 Thread Patrick Wendell
This vote passes with 5 +1 votes (3 binding) and no 0 or -1 votes. +1 Votes: Krishna Sankar Sean Owen* Chip Senkbeil Matei Zaharia* Patrick Wendell* 0 Votes: (none) -1 Votes: (none) On Fri, Feb 6, 2015 at 5:12 PM, Patrick Wendell wrote: > I'll add a +1 as well. > > On Fri, Feb 6

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-08 Thread Patrick Wendell
I'll add a +1 as well. On Fri, Feb 6, 2015 at 2:38 PM, Matei Zaharia wrote: > +1 > > Tested on Mac OS X. > > Matei > > >> On Feb 2, 2015, at 8:57 PM, Patrick Wendell wrote: >> >> Please vote on releasing the following candidate as Apache Spark versio

[jira] [Commented] (SPARK-5524) Remove messy dependencies to log4j

2015-02-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1430#comment-1430 ] Patrick Wendell commented on SPARK-5524: [~nchammas] I don't thin

[jira] [Updated] (SPARK-5524) Remove messy dependencies to log4j

2015-02-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5524: --- Component/s: (was: Build) > Remove messy dependencies to lo

[jira] [Updated] (SPARK-5524) Remove messy dependencies to log4j

2015-02-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5524: --- Component/s: Spark Core > Remove messy dependencies to lo

[jira] [Updated] (SPARK-5662) Flaky test: org.apache.spark.streaming.kafka.KafkaDirectStreamSuite.multi topic stream

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5662: --- Priority: Critical (was: Major) > Flaky t

[jira] [Created] (SPARK-5662) Flaky test: org.apache.spark.streaming.kafka.KafkaDirectStreamSuite.multi topic stream

2015-02-06 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5662: -- Summary: Flaky test: org.apache.spark.streaming.kafka.KafkaDirectStreamSuite.multi topic stream Key: SPARK-5662 URL: https://issues.apache.org/jira/browse/SPARK-5662

[jira] [Resolved] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5388. Resolution: Fixed Fix Version/s: 1.3.0 > Provide a stable application submiss

[jira] [Updated] (SPARK-5659) Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5659: --- Labels: flaky-test (was: ) > Flaky Test: org.apache.spark.streaming.ReceiverSuite.bl

[jira] [Updated] (SPARK-5659) Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5659: --- Priority: Critical (was: Major) > Flaky Test: org.apache.spark.streaming.ReceiverSuite.bl

[jira] [Created] (SPARK-5659) Flaky Test: org.apache.spark.streaming.ReceiverSuite.block

2015-02-06 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5659: -- Summary: Flaky Test: org.apache.spark.streaming.ReceiverSuite.block Key: SPARK-5659 URL: https://issues.apache.org/jira/browse/SPARK-5659 Project: Spark

[jira] [Updated] (SPARK-4874) Report number of records read/written in a task

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4874: --- Component/s: Web UI Spark Core > Report number of records read/written i

[jira] [Resolved] (SPARK-4874) Report number of records read/written in a task

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4874. Resolution: Fixed Fix Version/s: 1.3.0 Target Version/s: 1.3.0 > Rep

Re: Improving metadata in Spark JIRA

2015-02-06 Thread Patrick Wendell
Per Nick's suggestion I added two components: 1. Spark Submit 2. Spark Scheduler I figured I would just add these since if we decide later we don't want them, we can simply merge them into Spark Core. On Fri, Feb 6, 2015 at 11:53 AM, Nicholas Chammas wrote: > Do we need some new components to b

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309825#comment-14309825 ] Patrick Wendell commented on SPARK-5388: One the boolean and numeric value

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309784#comment-14309784 ] Patrick Wendell commented on SPARK-5388: On DELETE, I'll defer to you g

[jira] [Commented] (SPARK-5654) Integrate SparkR into Apache Spark

2015-02-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309759#comment-14309759 ] Patrick Wendell commented on SPARK-5654: It's a fair point to ask about

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308635#comment-14308635 ] Patrick Wendell commented on SPARK-5388: I think it's reasonable to u

[jira] [Commented] (SPARK-5607) NullPointerException in objenesis

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308493#comment-14308493 ] Patrick Wendell commented on SPARK-5607: I've reverted my patch since it

[jira] [Reopened] (SPARK-5607) NullPointerException in objenesis

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-5607: > NullPointerException in objenesis > - > >

[jira] [Commented] (SPARK-5626) Spurious test failures due to NullPointerException in EasyMock test code

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308482#comment-14308482 ] Patrick Wendell commented on SPARK-5626: [~joshrosen] I may have caused thi

[jira] [Resolved] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5557. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Patrick Wendell > Serv

[jira] [Commented] (SPARK-5607) NullPointerException in objenesis

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308423#comment-14308423 ] Patrick Wendell commented on SPARK-5607: This may have actually caused more o

[jira] [Commented] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308391#comment-14308391 ] Patrick Wendell commented on SPARK-5557: I'm sorry that this affecte

[jira] [Comment Edited] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308391#comment-14308391 ] Patrick Wendell edited comment on SPARK-5557 at 2/6/15 1:0

Re: PSA: Maven supports parallel builds

2015-02-05 Thread Patrick Wendell
I've done this in the past, but back when I wasn't using Zinc it didn't make a big difference. It's worth doing this in our jenkins environment though. - Patrick On Thu, Feb 5, 2015 at 4:52 PM, Dirceu Semighini Filho wrote: > Thanks Nicholas, I didn't knew this. > > 2015-02-05 22:16 GMT-02:00 Ni

[jira] [Updated] (SPARK-5557) Servlet API classes now missing after jetty shading

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5557: --- Summary: Servlet API classes now missing after jetty shading (was: spark-shell failed to

[jira] [Commented] (SPARK-5557) spark-shell failed to start

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308346#comment-14308346 ] Patrick Wendell commented on SPARK-5557: I can send a fix for this shortly

[jira] [Resolved] (SPARK-5474) curl should support URL redirection in build/mvn

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5474. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Guoqiang Li > curl sho

[jira] [Resolved] (SPARK-5548) Flaky test: org.apache.spark.util.AkkaUtilsSuite.remote fetch ssl on - untrusted server

2015-02-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5548. Resolution: Fixed Fix Version/s: 1.3.0 > Flaky t

[jira] [Resolved] (SPARK-5607) NullPointerException in objenesis

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5607. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Patrick Wendell

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Assignee: shengli > Automatically provide sqlContext in Spark sh

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Priority: Blocker (was: Critical) > Automatically provide sqlContext in Spark sh

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Assignee: (was: Patrick Wendell) > Automatically provide sqlContext in Spark sh

[jira] [Resolved] (SPARK-5411) Allow SparkListeners to be specified in SparkConf and loaded when creating SparkContext

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5411. Resolution: Fixed Fix Version/s: 1.3.0 > Allow SparkListeners to be specified

Re: multi-line comment style

2015-02-04 Thread Patrick Wendell
Personally I have no opinion, but agree it would be nice to standardize. - Patrick On Wed, Feb 4, 2015 at 1:58 PM, Sean Owen wrote: > One thing Marcelo pointed out to me is that the // style does not > interfere with commenting out blocks of code with /* */, which is a > small good thing. I am a

[jira] [Updated] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5594: --- Priority: Critical (was: Major) > SparkException: Failed to get broadcast (TorrentBroadc

[jira] [Commented] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-02-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305657#comment-14305657 ] Patrick Wendell commented on SPARK-5594: I've seen this occasionall

Re: 1.2.1-rc3 - Avro input format for Hadoop 2 broken/fix?

2015-02-04 Thread Patrick Wendell
Hi Markus, That won't be included in 1.2.1 most likely because the release votes have already started, and at that point we don't hold the release except for major regression issues from 1.2.0. However, if this goes through we can backport it into the 1.2 branch and it will end up in a future main

[jira] [Created] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-03 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5586: -- Summary: Automatically provide sqlContext in Spark shell Key: SPARK-5586 URL: https://issues.apache.org/jira/browse/SPARK-5586 Project: Spark Issue Type

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Fix Version/s: (was: 1.3.0) > Automatically provide sqlContext in Spark sh

[jira] [Updated] (SPARK-5586) Automatically provide sqlContext in Spark shell

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5586: --- Priority: Critical (was: Major) > Automatically provide sqlContext in Spark sh

[jira] [Resolved] (SPARK-5341) Support maven coordinates in spark-shell and spark-submit

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5341. Resolution: Fixed Fix Version/s: 1.3.0 > Support maven coordinates in spark-sh

[jira] [Updated] (SPARK-5341) Support maven coordinates in spark-shell and spark-submit

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5341: --- Assignee: Burak Yavuz > Support maven coordinates in spark-shell and spark-sub

[jira] [Updated] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5585: --- Priority: Critical (was: Major) > Flaky test: Python regress

[jira] [Updated] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5585: --- Labels: flaky-test (was: ) > Flaky test: Python regress

[jira] [Created] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5585: -- Summary: Flaky test: Python regression Key: SPARK-5585 URL: https://issues.apache.org/jira/browse/SPARK-5585 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-5585) Flaky test: Python regression

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5585: --- Affects Version/s: 1.3.0 > Flaky test: Python regress

[jira] [Updated] (SPARK-5140) Two RDDs which are scheduled concurrently should be able to wait on parent in all cases

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5140: --- Fix Version/s: (was: 1.2.1) (was: 1.3.0) > Two RDDs which

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304010#comment-14304010 ] Patrick Wendell commented on SPARK-5388: The intention for this is really jus

[ANNOUNCE] branch-1.3 has been cut

2015-02-03 Thread Patrick Wendell
Hey All, Just wanted to announce that we've cut the 1.3 branch which will become the 1.3 release after community testing. There are still some features that will go in (in higher level libraries, and some stragglers in spark core), but overall this indicates the end of major feature development f

[jira] [Created] (SPARK-5569) Checkpoints cannot reference classes defined outside of Spark's assembly

2015-02-03 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5569: -- Summary: Checkpoints cannot reference classes defined outside of Spark's assembly Key: SPARK-5569 URL: https://issues.apache.org/jira/browse/SPARK-5569 Pr

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302999#comment-14302999 ] Patrick Wendell commented on SPARK-4550: The doc alludes to having to (at

[jira] [Resolved] (SPARK-5542) Decouple publishing, packaging, and tagging in release script

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5542. Resolution: Fixed Fix Version/s: 1.3.0 > Decouple publishing, packaging, and tagg

[VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-02 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.2.1! The tag to be voted on is v1.2.1-rc3 (commit b6eaf77): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=b6eaf77d4332bfb0a698849b1f5f917d20d70e97 The release files, including signatures, digests, etc. can

[RESULT] [VOTE] Release Apache Spark 1.2.1 (RC2)

2015-02-02 Thread Patrick Wendell
This is cancelled in favor of RC2. On Mon, Feb 2, 2015 at 8:50 PM, Patrick Wendell wrote: > The windows issue reported only affects actually running Spark on > Windows (not job submission). However, I agree it's worth cutting a > new RC. I'm going to cancel this vote and

Re: [VOTE] Release Apache Spark 1.2.1 (RC2)

2015-02-02 Thread Patrick Wendell
The windows issue reported only affects actually running Spark on Windows (not job submission). However, I agree it's worth cutting a new RC. I'm going to cancel this vote and propose RC3 with a single additional patch. Let's try to vote that through so we can ship Spark 1.2.1. - Patrick On Sat,

[jira] [Created] (SPARK-5548) Flaky test: org.apache.spark.util.AkkaUtilsSuite.remote fetch ssl on - untrusted server

2015-02-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5548: -- Summary: Flaky test: org.apache.spark.util.AkkaUtilsSuite.remote fetch ssl on - untrusted server Key: SPARK-5548 URL: https://issues.apache.org/jira/browse/SPARK-5548

Temporary jenkins issue

2015-02-02 Thread Patrick Wendell
Hey All, I made a change to the Jenkins configuration that caused most builds to fail (attempting to enable a new plugin), I've reverted the change effective about 10 minutes ago. If you've seen recent build failures like below, this was caused by that change. Sorry about that. ERROR: Publi

[jira] [Updated] (SPARK-4508) Native Date type for SQL92 Date

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4508: --- Fix Version/s: (was: 1.3.0) > Native Date type for SQL92 D

[jira] [Reopened] (SPARK-4508) Native Date type for SQL92 Date

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-4508: This has caused several date-related test failures in the master and pull request builds, so

[jira] [Created] (SPARK-5542) Decouple publishing, packaging, and tagging in release script

2015-02-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5542: -- Summary: Decouple publishing, packaging, and tagging in release script Key: SPARK-5542 URL: https://issues.apache.org/jira/browse/SPARK-5542 Project: Spark

[jira] [Updated] (SPARK-5195) when hive table is query with alias the cache data lose effectiveness.

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5195: --- Fix Version/s: (was: 1.2.1) > when hive table is query with alias the cache data l

[jira] [Commented] (SPARK-5195) when hive table is query with alias the cache data lose effectiveness.

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302559#comment-14302559 ] Patrick Wendell commented on SPARK-5195: I reverted this from Spark 1.2. &g

[jira] [Updated] (SPARK-5541) Allow running Maven or SBT in run-tests

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5541: --- Summary: Allow running Maven or SBT in run-tests (was: Allow running Maven or SBT in the

[jira] [Created] (SPARK-5541) Allow running Maven or SBT in the Spark build

2015-02-02 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-5541: -- Summary: Allow running Maven or SBT in the Spark build Key: SPARK-5541 URL: https://issues.apache.org/jira/browse/SPARK-5541 Project: Spark Issue Type

[jira] [Commented] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302326#comment-14302326 ] Patrick Wendell commented on SPARK-4550: Yeah, this is a good idea. I don&#

[jira] [Updated] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4550: --- Target Version/s: 1.4.0 > In sort-based shuffle, store map outputs in serialized f

[jira] [Updated] (SPARK-4550) In sort-based shuffle, store map outputs in serialized form

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4550: --- Priority: Critical (was: Major) > In sort-based shuffle, store map outputs in seriali

[jira] [Commented] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302290#comment-14302290 ] Patrick Wendell commented on SPARK-3778: /cc [~hshreedharan] > newAPIHa

[jira] [Updated] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3778: --- Priority: Blocker (was: Critical) > newAPIHadoopRDD doesn't properly pass credent

Re: Spark Master Maven with YARN build is broken

2015-02-02 Thread Patrick Wendell
It's my fault, I'm sending a hot fix now. On Mon, Feb 2, 2015 at 1:44 PM, Nicholas Chammas wrote: > https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/ > > Is this is a known issue? It seems to have been broken since last nigh

[jira] [Comment Edited] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302006#comment-14302006 ] Patrick Wendell edited comment on SPARK-5388 at 2/2/15 9:5

[jira] [Commented] (SPARK-5388) Provide a stable application submission gateway in standalone cluster mode

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302006#comment-14302006 ] Patrick Wendell commented on SPARK-5388: Hey [~vanzin] and [~tigerquoll],

[jira] [Resolved] (SPARK-5492) Thread statistics can break with older Hadoop versions

2015-02-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5492. Resolution: Fixed Fix Version/s: 1.3.0 Thanks, Sandy. > Thread statistics can br

Re: Questions about Spark standalone resource scheduler

2015-02-02 Thread Patrick Wendell
Hey Jerry, I think standalone mode will still add more features over time, but the goal isn't really for it to become equivalent to what Mesos/YARN are today. Or at least, I doubt Spark Standalone will ever attempt to manage _other_ frameworks outside of Spark and become a general purpose resource

Re: Questions about Spark standalone resource scheduler

2015-02-02 Thread Patrick Wendell
Hey Jerry, I think standalone mode will still add more features over time, but the goal isn't really for it to become equivalent to what Mesos/YARN are today. Or at least, I doubt Spark Standalone will ever attempt to manage _other_ frameworks outside of Spark and become a general purpose resource

[jira] [Resolved] (SPARK-5478) Add miss right parenthesis in Stage page Pending stages label

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5478. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Saisai Shao > Add m

[jira] [Updated] (SPARK-5478) Add miss right parenthesis in Stage page Pending stages label

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5478: --- Affects Version/s: 1.3.0 > Add miss right parenthesis in Stage page Pending stages la

[jira] [Commented] (SPARK-4349) Spark driver hangs on sc.parallelize() if exception is thrown during serialization

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300913#comment-14300913 ] Patrick Wendell commented on SPARK-4349: [~mcheah] - minor, but when we clos

[jira] [Reopened] (SPARK-4349) Spark driver hangs on sc.parallelize() if exception is thrown during serialization

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-4349: > Spark driver hangs on sc.parallelize() if exception is thrown during > seriali

[jira] [Resolved] (SPARK-4349) Spark driver hangs on sc.parallelize() if exception is thrown during serialization

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4349. Resolution: Duplicate > Spark driver hangs on sc.parallelize() if exception is thr

[jira] [Updated] (SPARK-5500) Document that feeding hadoopFile into a shuffle operation will cause problems

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5500: --- Priority: Critical (was: Major) > Document that feeding hadoopFile into a shuffle operat

[jira] [Updated] (SPARK-1517) Publish nightly snapshots of documentation, maven artifacts, and binary builds

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1517: --- Assignee: Nicholas Chammas > Publish nightly snapshots of documentation, maven artifacts,

[jira] [Commented] (SPARK-5492) Thread statistics can break with older Hadoop versions

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300905#comment-14300905 ] Patrick Wendell commented on SPARK-5492: [~sandyr] I share your confusion, S

[jira] [Updated] (SPARK-5508) [hive context] Unable to query array once saved as parquet

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5508: --- Component/s: (was: spark sql) SQL > [hive context] Unable to query ar

[jira] [Updated] (SPARK-5508) [hive context] Unable to query array once saved as parquet

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5508: --- Component/s: (was: Spark Core) spark sql > [hive context] Unable

[jira] [Resolved] (SPARK-5353) Log failures in ExceutorClassLoader

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5353. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Tobias Schlatter >

[jira] [Commented] (SPARK-5515) Build fails with spark-ganglia-lgpl profile

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300894#comment-14300894 ] Patrick Wendell commented on SPARK-5515: [~andrewor14] is this fixed

[jira] [Resolved] (SPARK-5208) Add more documentation to Netty-based configs

2015-02-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5208. Resolution: Won't Fix > Add more documentation to Netty-based

<    6   7   8   9   10   11   12   13   14   15   >