[jira] [Updated] (SPARK-2630) Input data size of CoalescedRDD is incorrect

2014-11-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2630: --- Fix Version/s: (was: 1.2.0) Input data size of CoalescedRDD is incorrect

[jira] [Updated] (SPARK-4384) Too many open files during sort in pyspark

2014-11-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4384: --- Priority: Blocker (was: Critical) Too many open files during sort in pyspark

Build break

2014-11-19 Thread Patrick Wendell
Hey All, Just a heads up. I merged this patch last night which caused the Spark build to break: https://github.com/apache/spark/commit/397d3aae5bde96b01b4968dde048b6898bb6c914 The patch itself was fine and previously had passed on Jenkins. The issue was that other intermediate changes merged

[jira] [Resolved] (SPARK-4017) Progress bar in console

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4017. Resolution: Fixed Fix Version/s: 1.2.0 Progress bar in console

[jira] [Updated] (SPARK-4377) ZooKeeperPersistenceEngine: java.lang.IllegalStateException: Trying to deserialize a serialized ActorRef without an ActorSystem in scope.

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4377: --- Target Version/s: 1.2.0 ZooKeeperPersistenceEngine: java.lang.IllegalStateException: Trying

[jira] [Resolved] (SPARK-4281) Yarn shuffle service jars need to include dependencies

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4281. Resolution: Fixed Fix Version/s: 1.2.0 Yarn shuffle service jars need to include

[jira] [Updated] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4452: --- Component/s: Spark Core Shuffle data structures can starve others on the same thread

[jira] [Reopened] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-4404: SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[jira] [Closed] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell closed SPARK-4404. -- Resolution: Fixed SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process

[jira] [Resolved] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4404. Resolution: Fixed SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub

[jira] [Closed] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell closed SPARK-4404. -- SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

[jira] [Resolved] (SPARK-4404) SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub-process ends

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4404. Resolution: Fixed SparkSubmitDriverBootstrapper should stop after its SparkSubmit sub

[jira] [Resolved] (SPARK-4441) Close Tachyon client when TachyonBlockManager is shut down

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4441. Resolution: Fixed Fix Version/s: 1.2.0 Close Tachyon client when

[jira] [Resolved] (SPARK-4432) Resource(InStream) is not closed in TachyonStore

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4432. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: shimingfei Resource

[jira] [Updated] (SPARK-4441) Close Tachyon client when TachyonBlockManager is shut down

2014-11-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4441: --- Assignee: shimingfei Close Tachyon client when TachyonBlockManager is shut down

[jira] [Updated] (SPARK-3962) Mark spark dependency as provided in external libraries

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3962: --- Issue Type: Bug (was: Improvement) Mark spark dependency as provided in external libraries

[jira] [Commented] (SPARK-3962) Mark spark dependency as provided in external libraries

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214480#comment-14214480 ] Patrick Wendell commented on SPARK-3962: I think this is causing the build to fail

[jira] [Updated] (SPARK-2811) update algebird to 0.8.1

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2811: --- Assignee: Adam Pingel update algebird to 0.8.1

[jira] [Resolved] (SPARK-2811) update algebird to 0.8.1

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2811. Resolution: Fixed Fix Version/s: 1.2.0 update algebird to 0.8.1

[jira] [Updated] (SPARK-4180) SparkContext constructor should throw exception if another SparkContext is already running

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4180: --- Fix Version/s: 1.2.0 SparkContext constructor should throw exception if another SparkContext

[jira] [Created] (SPARK-4466) Provide support for publishing Scala 2.11 artifacts to Maven

2014-11-17 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4466: -- Summary: Provide support for publishing Scala 2.11 artifacts to Maven Key: SPARK-4466 URL: https://issues.apache.org/jira/browse/SPARK-4466 Project: Spark

[jira] [Updated] (SPARK-4466) Provide support for publishing Scala 2.11 artifacts to Maven

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4466: --- Assignee: Patrick Wendell Provide support for publishing Scala 2.11 artifacts to Maven

[jira] [Updated] (SPARK-4286) Support External Shuffle Service with Mesos integration

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4286: --- Assignee: Timothy Chen Support External Shuffle Service with Mesos integration

[jira] [Resolved] (SPARK-4466) Provide support for publishing Scala 2.11 artifacts to Maven

2014-11-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4466. Resolution: Fixed Fix Version/s: 1.2.0 Provide support for publishing Scala 2.11

Re: [VOTE] Release Apache Spark 1.1.1 (RC1)

2014-11-17 Thread Patrick Wendell
Hey Kevin, If you are upgrading from 1.0.X to 1.1.X checkout the upgrade notes here [1] - it could be that default changes caused a regression for your workload. Do you still see a regression if you restore the configuration changes? It's great to hear specifically about issues like this, so

[jira] [Commented] (SPARK-4399) Support multiple cloud providers

2014-11-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214153#comment-14214153 ] Patrick Wendell commented on SPARK-4399: I think this might actually be out

[jira] [Created] (SPARK-4445) Don't display storage level in toDebugString unless RDD is persisted

2014-11-16 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4445: -- Summary: Don't display storage level in toDebugString unless RDD is persisted Key: SPARK-4445 URL: https://issues.apache.org/jira/browse/SPARK-4445 Project

[jira] [Updated] (SPARK-4445) Don't display storage level in toDebugString unless RDD is persisted

2014-11-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4445: --- Issue Type: Bug (was: Improvement) Don't display storage level in toDebugString unless RDD

Re: mvn or sbt for studying and developing Spark?

2014-11-16 Thread Patrick Wendell
Neither is strictly optimal which is why we ended up supporting both. Our reference build for packaging is Maven so you are less likely to run into unexpected dependency issues, etc. Many developers use sbt as well. It's somewhat religion and the best thing might be to try both and see which you

Re: Has anyone else observed this build break?

2014-11-15 Thread Patrick Wendell
Server VM (build 24.60-b09, mixed mode) Let me see if the problem can be solved upstream in HBase hbase-annotations module. Cheers On Fri, Nov 14, 2014 at 12:32 PM, Patrick Wendell pwend...@gmail.com wrote: I think in this case we can probably just drop that dependency, so there is a simpler

Has anyone else observed this build break?

2014-11-14 Thread Patrick Wendell
A recent patch broke clean builds for me, I am trying to see how widespread this issue is and whether we need to revert the patch. The error I've seen is this when building the examples project: spark-examples_2.10: Could not resolve dependencies for project

Re: Has anyone else observed this build break?

2014-11-14 Thread Patrick Wendell
A work around for this fix is identified here: http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html However, if this affects more users I'd prefer to just fix it properly in our build. On Fri, Nov 14, 2014 at 12:17 PM, Patrick Wendell pwend...@gmail.com wrote

Re: Has anyone else observed this build break?

2014-11-14 Thread Patrick Wendell
this can fix it? Thanks, Hari On Fri, Nov 14, 2014 at 12:21 PM, Patrick Wendell pwend...@gmail.com wrote: A work around for this fix is identified here: http://dbknickerbocker.blogspot.com/2013/04/simple-fix-to-missing-toolsjar-in-jdk.html However, if this affects more users I'd prefer

Re: toLocalIterator in Spark 1.0.0

2014-11-13 Thread Patrick Wendell
It looks like you are trying to directly import the toLocalIterator function. You can't import functions, it should just appear as a method of an existing RDD if you have one. - Patrick On Thu, Nov 13, 2014 at 10:21 PM, Deep Pradhan pradhandeep1...@gmail.com wrote: Hi, I am using Spark 1.0.0

[jira] [Created] (SPARK-4376) Put external modules behind build profiles

2014-11-12 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4376: -- Summary: Put external modules behind build profiles Key: SPARK-4376 URL: https://issues.apache.org/jira/browse/SPARK-4376 Project: Spark Issue Type

[jira] [Comment Edited] (SPARK-4375) Assembly built with Maven is missing most of repl classes

2014-11-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209305#comment-14209305 ] Patrick Wendell edited comment on SPARK-4375 at 11/13/14 5:37 AM

[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes

2014-11-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209305#comment-14209305 ] Patrick Wendell commented on SPARK-4375: Hey Sandy, What about the following

[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes

2014-11-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209314#comment-14209314 ] Patrick Wendell commented on SPARK-4375: One thing we could add onto that to make

[jira] [Updated] (SPARK-2450) Provide link to YARN executor logs on UI

2014-11-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2450: --- Assignee: Kostas Sakellis Provide link to YARN executor logs on UI

[jira] [Commented] (SPARK-1296) Make RDDs Covariant

2014-11-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209361#comment-14209361 ] Patrick Wendell commented on SPARK-1296: Because it's not possible to do without

[jira] [Commented] (SPARK-4375) Assembly built with Maven is missing most of repl classes

2014-11-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209416#comment-14209416 ] Patrick Wendell commented on SPARK-4375: So my favorite option now is just moving

Re: [NOTICE] [BUILD] Minor changes to Spark's build

2014-11-12 Thread Patrick Wendell
, IntelliJ will temporarily think things like the Kafka module are being removed. Say 'no' when it asks if you want to remove them. - Can we go straight to Scala 2.11.4? On Wed, Nov 12, 2014 at 5:47 AM, Patrick Wendell pwend...@gmail.com wrote: Hey All, I've just merged a patch that adds

Re: [NOTICE] [BUILD] Minor changes to Spark's build

2014-11-12 Thread Patrick Wendell
scrapco...@gmail.com wrote: One thing we can do it is print a helpful error and break. I don't know about how this can be done, but since now I can write groovy inside maven build so we have more control. (Yay!!) Prashant Sharma On Thu, Nov 13, 2014 at 12:05 PM, Patrick Wendell pwend

Re: [NOTICE] [BUILD] Minor changes to Spark's build

2014-11-12 Thread Patrick Wendell
: Currently there are no mandatory profiles required to build Spark. I.e. mvn package just works. It seems sad that we would need to break this. On Wed, Nov 12, 2014 at 10:59 PM, Patrick Wendell pwend...@gmail.com wrote: I think printing an error that says -Pscala-2.10 must be enabled

[jira] [Resolved] (SPARK-1812) Support cross-building with Scala 2.11

2014-11-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1812. Resolution: Fixed Fix Version/s: 1.2.0 The initial version of this patch has been

[jira] [Created] (SPARK-4356) Test Scala 2.11 in maven

2014-11-11 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4356: -- Summary: Test Scala 2.11 in maven Key: SPARK-4356 URL: https://issues.apache.org/jira/browse/SPARK-4356 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-4356) Test Scala 2.11 in maven

2014-11-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4356: --- Component/s: Project Infra Test Scala 2.11 in maven

[jira] [Updated] (SPARK-4356) Test Scala 2.11 on Jenkins

2014-11-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4356: --- Summary: Test Scala 2.11 on Jenkins (was: Test Scala 2.11 in maven) Test Scala 2.11

[jira] [Updated] (SPARK-4356) Test Scala 2.11 on Jenkins

2014-11-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4356: --- Issue Type: Improvement (was: Bug) Test Scala 2.11 on Jenkins

[jira] [Updated] (SPARK-4356) Test Scala 2.11 on Jenkins

2014-11-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4356: --- Priority: Critical (was: Major) Test Scala 2.11 on Jenkins

[jira] [Created] (SPARK-4357) Modify release publishing to work with Scala 2.11

2014-11-11 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4357: -- Summary: Modify release publishing to work with Scala 2.11 Key: SPARK-4357 URL: https://issues.apache.org/jira/browse/SPARK-4357 Project: Spark Issue

[jira] [Updated] (SPARK-4357) Modify release publishing to work with Scala 2.11

2014-11-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4357: --- Component/s: Project Infra Modify release publishing to work with Scala 2.11

Re: JIRA + PR backlog

2014-11-11 Thread Patrick Wendell
I wonder if we should be linking to that dashboard somewhere from our official docs or the wiki... On Tue, Nov 11, 2014 at 12:23 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Yeah, kudos to Josh for putting that together. On Tue, Nov 11, 2014 at 3:26 AM, Yu Ishikawa

[NOTICE] [BUILD] Minor changes to Spark's build

2014-11-11 Thread Patrick Wendell
Hey All, I've just merged a patch that adds support for Scala 2.11 which will have some minor implications for the build. These are due to the complexities of supporting two versions of Scala in a single project. 1. The JDBC server will now require a special flag to build -Phive-thriftserver on

Re: Still struggling with building documentation

2014-11-11 Thread Patrick Wendell
The doc build appears to be broken in master. We'll get it patched up before the release: https://issues.apache.org/jira/browse/SPARK-4326 On Tue, Nov 11, 2014 at 10:50 AM, Alessandro Baretta alexbare...@gmail.com wrote: Nichols and Patrick, Thanks for your help, but, no, it still does not

Re: Spark and Play

2014-11-11 Thread Patrick Wendell
Hi There, Because Akka versions are not binary compatible with one another, it might not be possible to integrate Play with Spark 1.1.0. - Patrick On Tue, Nov 11, 2014 at 8:21 AM, Akshat Aranya aara...@gmail.com wrote: Hi, Sorry if this has been asked before; I didn't find a satisfactory

[jira] [Resolved] (SPARK-4312) bash can't `die`

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4312. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Kousuke Saruta bash can't

[jira] [Resolved] (SPARK-4230) Doc for spark.default.parallelism is incorrect

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4230. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Sandy Ryza Doc

[jira] [Resolved] (SPARK-1297) Upgrade HBase dependency to 0.98.0

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1297. Resolution: Fixed Fix Version/s: 1.2.0 Upgrade HBase dependency to 0.98.0

[jira] [Commented] (SPARK-2703) Make Tachyon related unit tests execute without deploying a Tachyon system locally.

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205486#comment-14205486 ] Patrick Wendell commented on SPARK-2703: FYI I had to revert this patch because

[jira] [Commented] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205520#comment-14205520 ] Patrick Wendell commented on SPARK-3461: I think [~sandyr] wanted to take a crack

[jira] [Commented] (SPARK-4331) Scalastyle doesn't work for the sources under hive's v0.12.0 and v0.13.1

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205795#comment-14205795 ] Patrick Wendell commented on SPARK-4331: This is going to be exacerbated

[jira] [Commented] (SPARK-4314) Exception throws when the upload intermediate file(_COPYING_ file) is read through hdfs interface

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205799#comment-14205799 ] Patrick Wendell commented on SPARK-4314: Can you add a filter that excludes

[jira] [Updated] (SPARK-4314) Exception throws when the upload intermediate file(_COPYING_ file) is read through hdfs interface

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4314: --- Component/s: Streaming Exception throws when the upload intermediate file(_COPYING_ file

[jira] [Comment Edited] (SPARK-4314) Exception throws when the upload intermediate file(_COPYING_ file) is read through hdfs interface

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205799#comment-14205799 ] Patrick Wendell edited comment on SPARK-4314 at 11/11/14 1:56 AM

[jira] [Updated] (SPARK-4314) Exception when textFileStream attempts to read deleted _COPYING_ file

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4314: --- Summary: Exception when textFileStream attempts to read deleted _COPYING_ file

[jira] [Updated] (SPARK-4335) Mima check misreporting for GraphX pull request

2014-11-10 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4335: --- Assignee: Prashant Sharma Mima check misreporting for GraphX pull request

[jira] [Resolved] (SPARK-3648) Provide a script for fetching remote PR's for review

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3648. Resolution: Won't Fix Provide a script for fetching remote PR's for review

[jira] [Reopened] (SPARK-1739) Close PR's after 30 days of inactivity

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-1739: I'd actually like to re-open this since I think there is still a need for it. Will update

[jira] [Updated] (SPARK-1739) Close PR's after 30 days of inactivity

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1739: --- Description: Sometimes PR's get abandoned if people aren't responsive to feedback or it just

[jira] [Updated] (SPARK-1739) Close PR's after 30 days of inactivity

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1739: --- Description: Sometimes PR's get abandoned if people aren't responsive to feedback or it just

[jira] [Updated] (SPARK-1739) Close PR's after 30 days of inactivity

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1739: --- Description: Sometimes PR's get abandoned if people aren't responsive to feedback or it just

[jira] [Updated] (SPARK-1739) Close PR's after 30 days of inactivity

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1739: --- Assignee: Josh Rosen (was: Patrick Wendell) Close PR's after 30 days of inactivity

[jira] [Updated] (SPARK-1739) Close PR's after 30 days of inactivity

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1739: --- Description: Sometimes PR's get abandoned if people aren't responsive to feedback or it just

[jira] [Resolved] (SPARK-971) Link to Confluence wiki from project website / documentation

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-971. --- Resolution: Fixed Assignee: Sean Owen https://cwiki.apache.org/confluence/display/SPARK

[jira] [Resolved] (SPARK-1344) Scala API docs for top methods

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1344. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Sean Owen Scala API docs

[jira] [Resolved] (SPARK-2083) Allow local task to retry after failure.

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2083. Resolution: Won't Fix Already supported through spark-context construction. Allow local

[jira] [Resolved] (SPARK-3191) Add explanation of supporting building spark with maven in http proxy environment

2014-11-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3191. Resolution: Won't Fix Hi there - I thought a bit more about this and I think we probably

Re: Should new YARN shuffle service work with yarn-alpha?

2014-11-08 Thread Patrick Wendell
refactoring needed? Either to support YARN alpha as a separate shuffle module, or sever this dependency? Of course this goes away when yarn-alpha goes away too. On Sat, Nov 8, 2014 at 7:45 AM, Patrick Wendell pwend...@gmail.com wrote: I bet it doesn't work. +1 on isolating it's inclusion

Re: Should new YARN shuffle service work with yarn-alpha?

2014-11-08 Thread Patrick Wendell
. That makes yarn-alpha work. I'll run tests and open a quick JIRA / PR for the change. On Sat, Nov 8, 2014 at 8:23 AM, Patrick Wendell pwend...@gmail.com wrote: This second error is something else. Maybe you are excluding network-shuffle instead of spark-network-yarn

[jira] [Resolved] (SPARK-4291) Drop Code from network module names

2014-11-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4291. Resolution: Fixed Fix Version/s: 1.2.0 Drop Code from network module names

Re: Should new YARN shuffle service work with yarn-alpha?

2014-11-07 Thread Patrick Wendell
I bet it doesn't work. +1 on isolating it's inclusion to only the newer YARN API's. - Patrick On Fri, Nov 7, 2014 at 11:43 PM, Sean Owen so...@cloudera.com wrote: I noticed that this doesn't compile: mvn -Pyarn-alpha -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package [error]

[jira] [Updated] (SPARK-4266) Avoid $$ JavaScript for StagePages with huge numbers of tables

2014-11-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4266: --- Priority: Critical (was: Major) Avoid $$ JavaScript for StagePages with huge numbers

[jira] [Commented] (SPARK-2447) Add common solution for sending upsert actions to HBase (put, deletes, and increment)

2014-11-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200634#comment-14200634 ] Patrick Wendell commented on SPARK-2447: Hey All, I have a question about

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-11-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201042#comment-14201042 ] Patrick Wendell commented on SPARK-3561: Hey [~ozhurakousky] - as I said before, I

Re: [VOTE] Designating maintainers for some Spark components

2014-11-06 Thread Patrick Wendell
I think new committers might or might not be maintainers (it would depend on the PMC vote). I don't think it would affect what you could merge, you can merge in any part of the source tree, you just need to get sign off if you want to touch a public API or make major architectural changes. Most

Re: [VOTE] Designating maintainers for some Spark components

2014-11-06 Thread Patrick Wendell
Hey Greg, Regarding subversion - I think the reference is to partial vs full committers here: https://subversion.apache.org/docs/community-guide/roles.html - Patrick On Thu, Nov 6, 2014 at 4:18 PM, Greg Stein gst...@gmail.com wrote: -1 (non-binding) This is an idea that runs COMPLETELY

Re: [VOTE] Designating maintainers for some Spark components

2014-11-06 Thread Patrick Wendell
In fact, if you look at the subversion commiter list, the majority of people here have commit access only for particular areas of the project: http://svn.apache.org/repos/asf/subversion/trunk/COMMITTERS On Thu, Nov 6, 2014 at 4:26 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Greg

Re: [VOTE] Designating maintainers for some Spark components

2014-11-05 Thread Patrick Wendell
I'm a +1 on this as well, I think it will be a useful model as we scale the project in the future and recognizes some informal process we have now. To respond to Sandy's comment: for changes that fall in between the component boundaries or are straightforward, my understanding of this model is

[jira] [Updated] (SPARK-2938) Support SASL authentication in Netty network module

2014-11-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2938: --- Priority: Blocker (was: Major) Support SASL authentication in Netty network module

[jira] [Resolved] (SPARK-4178) Hadoop input metrics ignore bytes read in RecordReader instantiation

2014-11-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4178. Resolution: Fixed Assignee: Sandy Ryza Hadoop input metrics ignore bytes read

branch-1.2 has been cut

2014-11-03 Thread Patrick Wendell
Hi All, I've just cut the release branch for Spark 1.2, consistent with then end of the scheduled feature window for the release. New commits to master will need to be explicitly merged into branch-1.2 in order to be in the release. This begins the transition into a QA period for Spark 1.2, with

[jira] [Commented] (SPARK-4180) SparkContext constructor should throw exception if another SparkContext is already running

2014-11-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193747#comment-14193747 ] Patrick Wendell commented on SPARK-4180: Yeah [~adav] just ran into an issue where

[jira] [Resolved] (SPARK-4183) Enable Netty-based BlockTransferService by default

2014-11-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4183. Resolution: Fixed Resolved a second time via: https://github.com/apache/spark/pull/3053

[jira] [Resolved] (SPARK-4200) akka.loglevel

2014-11-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4200. Resolution: Invalid Hi There, For issues like this do you mind e-mailing the spark user

[jira] [Resolved] (SPARK-4177) update build doc for already supporting hive 13 in jdbc/cli

2014-11-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4177. Resolution: Fixed Assignee: wangfei update build doc for already supporting hive 13

[jira] [Updated] (SPARK-3572) Internal API for User-Defined Types

2014-11-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3572: --- Summary: Internal API for User-Defined Types (was: Support register UserType in SQL

Re: sbt scala compiler crashes on spark-sql

2014-11-02 Thread Patrick Wendell
Does this happen if you clean and recompile? I've seen failures on and off, but haven't been able to find one that I could reproduce from a clean build such that we could hand it to the scala team. - Patrick On Sun, Nov 2, 2014 at 7:25 PM, Imran Rashid im...@therashids.com wrote: I'm finding

Re: sbt scala compiler crashes on spark-sql

2014-11-02 Thread Patrick Wendell
versa. A clean rebuild can always solve this. On Mon, Nov 3, 2014 at 11:28 AM, Patrick Wendell pwend...@gmail.com wrote: Does this happen if you clean and recompile? I've seen failures on and off, but haven't been able to find one that I could reproduce from a clean build such that we

[jira] [Resolved] (SPARK-4183) Enable Netty-based BlockTransferService by default

2014-11-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4183. Resolution: Fixed Fix Version/s: 1.2.0 Enable Netty-based BlockTransferService

<    12   13   14   15   16   17   18   19   20   21   >