[jira] [Resolved] (SPARK-4242) Add SASL to external shuffle service

2014-11-06 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4242. --- Resolution: Fixed Fix Version/s: 1.2.0 Add SASL to external shuffle service

[jira] [Created] (SPARK-4264) SQL HashJoin induces refCnt = 0 error in ShuffleBlockFetcherIterator

2014-11-05 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4264: - Summary: SQL HashJoin induces refCnt = 0 error in ShuffleBlockFetcherIterator Key: SPARK-4264 URL: https://issues.apache.org/jira/browse/SPARK-4264 Project: Spark

[jira] [Commented] (SPARK-2468) Netty-based block server / client module

2014-11-05 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199911#comment-14199911 ] Aaron Davidson commented on SPARK-2468: --- This could be due to the netty transfer

[jira] [Comment Edited] (SPARK-2468) Netty-based block server / client module

2014-11-05 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199911#comment-14199911 ] Aaron Davidson edited comment on SPARK-2468 at 11/6/14 6:57 AM:

[jira] [Created] (SPARK-4236) External shuffle service must cleanup its shuffle files

2014-11-04 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4236: - Summary: External shuffle service must cleanup its shuffle files Key: SPARK-4236 URL: https://issues.apache.org/jira/browse/SPARK-4236 Project: Spark

[jira] [Created] (SPARK-4238) Perform network-level retry of shuffle file fetches

2014-11-04 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4238: - Summary: Perform network-level retry of shuffle file fetches Key: SPARK-4238 URL: https://issues.apache.org/jira/browse/SPARK-4238 Project: Spark Issue

[jira] [Created] (SPARK-4242) Add SASL to external shuffle service

2014-11-04 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4242: - Summary: Add SASL to external shuffle service Key: SPARK-4242 URL: https://issues.apache.org/jira/browse/SPARK-4242 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4198) Refactor BlockManager's doPut and doGetLocal into smaller pieces

2014-11-02 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4198: - Summary: Refactor BlockManager's doPut and doGetLocal into smaller pieces Key: SPARK-4198 URL: https://issues.apache.org/jira/browse/SPARK-4198 Project: Spark

[jira] [Created] (SPARK-4183) Enable Netty-based BlockTransferService by default

2014-11-01 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4183: - Summary: Enable Netty-based BlockTransferService by default Key: SPARK-4183 URL: https://issues.apache.org/jira/browse/SPARK-4183 Project: Spark Issue

[jira] [Updated] (SPARK-4187) External shuffle service should not use Java serializer

2014-11-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-4187: -- Component/s: Spark Core Affects Version/s: 1.2.0 External shuffle service should not

[jira] [Created] (SPARK-4187) External shuffle service should not use Java serializer

2014-11-01 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4187: - Summary: External shuffle service should not use Java serializer Key: SPARK-4187 URL: https://issues.apache.org/jira/browse/SPARK-4187 Project: Spark

[jira] [Created] (SPARK-4188) Shuffle fetches should be retried at a lower level

2014-11-01 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4188: - Summary: Shuffle fetches should be retried at a lower level Key: SPARK-4188 URL: https://issues.apache.org/jira/browse/SPARK-4188 Project: Spark Issue

[jira] [Created] (SPARK-4189) FileSegmentManagedBuffer should have a configurable memory map threshold

2014-11-01 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4189: - Summary: FileSegmentManagedBuffer should have a configurable memory map threshold Key: SPARK-4189 URL: https://issues.apache.org/jira/browse/SPARK-4189 Project:

[jira] [Resolved] (SPARK-4084) Reuse sort key in Sorter

2014-10-28 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-4084. --- Resolution: Fixed Fix Version/s: 1.2.0 Reuse sort key in Sorter

[jira] [Created] (SPARK-4106) Shuffle write and spill to disk metrics are not incorrect

2014-10-27 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-4106: - Summary: Shuffle write and spill to disk metrics are not incorrect Key: SPARK-4106 URL: https://issues.apache.org/jira/browse/SPARK-4106 Project: Spark

[jira] [Assigned] (SPARK-3994) countByKey / countByValue do not go through Aggregator

2014-10-17 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson reassigned SPARK-3994: - Assignee: Aaron Davidson countByKey / countByValue do not go through Aggregator

[jira] [Created] (SPARK-3994) countByKey / countByValue do not go through Aggregator

2014-10-17 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-3994: - Summary: countByKey / countByValue do not go through Aggregator Key: SPARK-3994 URL: https://issues.apache.org/jira/browse/SPARK-3994 Project: Spark Issue

[jira] [Created] (SPARK-3950) Completed time is blank for some successful tasks

2014-10-14 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-3950: - Summary: Completed time is blank for some successful tasks Key: SPARK-3950 URL: https://issues.apache.org/jira/browse/SPARK-3950 Project: Spark Issue

[jira] [Commented] (SPARK-3950) Completed time is blank for some successful tasks

2014-10-14 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171563#comment-14171563 ] Aaron Davidson commented on SPARK-3950: --- [~andrewor14] Completed time is blank for

[jira] [Updated] (SPARK-3921) WorkerWatcher in Standalone mode fail to come up due to invalid workerUrl

2014-10-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-3921: -- Description: As of [this

[jira] [Commented] (SPARK-3923) All Standalone Mode services time out with each other

2014-10-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169044#comment-14169044 ] Aaron Davidson commented on SPARK-3923: --- I did a little digging hoping to find some

[jira] [Commented] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2014-10-10 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167647#comment-14167647 ] Aaron Davidson commented on SPARK-3889: --- Sorry, it was not linked:

[jira] [Updated] (SPARK-3796) Create shuffle service for external block storage

2014-10-09 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-3796: -- Description: This task will be broken up into two parts -- the first, being to refactor our

[jira] [Created] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2014-10-09 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-3889: - Summary: JVM dies with SIGBUS, resulting in ConnectionManager failed ACK Key: SPARK-3889 URL: https://issues.apache.org/jira/browse/SPARK-3889 Project: Spark

[jira] [Updated] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2014-10-09 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-3889: -- Description: Here's the first part of the core dump, possibly caused by a job which shuffles a

[jira] [Commented] (SPARK-3805) Enable Standalone worker cleanup by default

2014-10-08 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164485#comment-14164485 ] Aaron Davidson commented on SPARK-3805: --- Upon further thought, I also agree with

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-10-04 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159343#comment-14159343 ] Aaron Davidson commented on SPARK-1860: --- Agreed, that sounds good. Would you or

[jira] [Resolved] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-10-03 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1860. --- Resolution: Fixed Fixed by mccheah in https://github.com/apache/spark/pull/2609 Standalone

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-10-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155303#comment-14155303 ] Aaron Davidson commented on SPARK-1860: --- The Worker itself is solely a Standalone

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-09-30 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152839#comment-14152839 ] Aaron Davidson commented on SPARK-1860: --- The Executor could clean up its own jars

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-09-30 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154228#comment-14154228 ] Aaron Davidson commented on SPARK-1860: --- Your logic SGTM, but I would add one

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-09-29 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152744#comment-14152744 ] Aaron Davidson commented on SPARK-1860: --- Note that there are two separate forms of

[jira] [Reopened] (SPARK-2973) Add a way to show tables without executing a job

2014-09-26 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson reopened SPARK-2973: --- Reopening this because sql(show tables).take(1) still starts a job. Add a way to show tables

[jira] [Updated] (SPARK-2973) Add a way to show tables without executing a job

2014-09-26 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-2973: -- Assignee: Michael Armbrust (was: Cheng Lian) Add a way to show tables without executing a job

[jira] [Commented] (SPARK-3267) Deadlock between ScalaReflectionLock and Data type initialization

2014-09-24 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146039#comment-14146039 ] Aaron Davidson commented on SPARK-3267: --- I don't have it anymore, unfortunately.

[jira] [Commented] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

2014-09-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143792#comment-14143792 ] Aaron Davidson commented on SPARK-3032: --- [~matei] any thoughts on this issue?

[jira] [Created] (SPARK-3267) Deadlock between ScalaReflectionLock and Data type initialization

2014-08-27 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-3267: - Summary: Deadlock between ScalaReflectionLock and Data type initialization Key: SPARK-3267 URL: https://issues.apache.org/jira/browse/SPARK-3267 Project: Spark

[jira] [Created] (SPARK-3236) Reading Parquet tables from Metastore mangles location

2014-08-26 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-3236: - Summary: Reading Parquet tables from Metastore mangles location Key: SPARK-3236 URL: https://issues.apache.org/jira/browse/SPARK-3236 Project: Spark Issue

[jira] [Resolved] (SPARK-3093) masterLock in Worker is no longer need

2014-08-18 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-3093. --- Resolution: Fixed Assignee: Chen Chao Target Version/s: 1.2.0

[jira] [Created] (SPARK-3029) Disable local execution of Spark jobs by default

2014-08-14 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-3029: - Summary: Disable local execution of Spark jobs by default Key: SPARK-3029 URL: https://issues.apache.org/jira/browse/SPARK-3029 Project: Spark Issue Type:

[jira] [Created] (SPARK-2973) Add a way to show tables without executing a job

2014-08-11 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2973: - Summary: Add a way to show tables without executing a job Key: SPARK-2973 URL: https://issues.apache.org/jira/browse/SPARK-2973 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-2936) Migrate Netty network module from Java to Scala

2014-08-10 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-2936. --- Resolution: Fixed Migrate Netty network module from Java to Scala

[jira] [Created] (SPARK-2949) SparkContext does not fate-share with ActorSystem

2014-08-09 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2949: - Summary: SparkContext does not fate-share with ActorSystem Key: SPARK-2949 URL: https://issues.apache.org/jira/browse/SPARK-2949 Project: Spark Issue

[jira] [Resolved] (SPARK-2557) createTaskScheduler should be consistent between local and local-n-failures

2014-08-01 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-2557. --- Resolution: Fixed Fix Version/s: 1.1.0 createTaskScheduler should be consistent

[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running applications

2014-07-28 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076403#comment-14076403 ] Aaron Davidson commented on SPARK-1860: --- There's not an easy way to tell if an

[jira] [Updated] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-07-28 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-1860: -- Description: The default values of the standalone worker cleanup code cleanup all application

[jira] [Commented] (SPARK-2707) Upgrade to Akka 2.3

2014-07-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14075652#comment-14075652 ] Aaron Davidson commented on SPARK-2707: --- It does sound mostly mechanical and I

[jira] [Commented] (SPARK-2707) Upgrade to Akka 2.3

2014-07-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14075744#comment-14075744 ] Aaron Davidson commented on SPARK-2707: --- That doesn't sound like a bad idea --

[jira] [Updated] (SPARK-1264) Documentation for setting heap sizes across all configurations

2014-07-24 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-1264: -- Assignee: (was: Aaron Davidson) Documentation for setting heap sizes across all

[jira] [Created] (SPARK-2660) Enable pretty-printing SchemaRDD Rows

2014-07-23 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2660: - Summary: Enable pretty-printing SchemaRDD Rows Key: SPARK-2660 URL: https://issues.apache.org/jira/browse/SPARK-2660 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2282) PySpark crashes if too many tasks complete quickly

2014-07-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070528#comment-14070528 ] Aaron Davidson commented on SPARK-2282: --- Great to hear! These files haven't been

[jira] [Commented] (SPARK-2282) PySpark crashes if too many tasks complete quickly

2014-07-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071192#comment-14071192 ] Aaron Davidson commented on SPARK-2282: --- [~pwendell] That would in general be the

[jira] [Comment Edited] (SPARK-1767) Prefer HDFS-cached replicas when scheduling data-local tasks

2014-07-21 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997761#comment-13997761 ] Aaron Davidson edited comment on SPARK-1767 at 7/21/14 7:46 PM:

[jira] [Commented] (SPARK-2282) PySpark crashes if too many tasks complete quickly

2014-07-20 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068083#comment-14068083 ] Aaron Davidson commented on SPARK-2282: --- Hey Ken, I created [PR

[jira] [Commented] (SPARK-2282) PySpark crashes if too many tasks complete quickly

2014-07-17 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065121#comment-14065121 ] Aaron Davidson commented on SPARK-2282: --- This problem does look identical. I think I

[jira] [Commented] (SPARK-2282) PySpark crashes if too many tasks complete quickly

2014-07-17 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065306#comment-14065306 ] Aaron Davidson commented on SPARK-2282: --- This problem is kinda silly because we're

[jira] [Created] (SPARK-2545) Add a diagnosis mode for closures to figure out what they're bringing in

2014-07-16 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2545: - Summary: Add a diagnosis mode for closures to figure out what they're bringing in Key: SPARK-2545 URL: https://issues.apache.org/jira/browse/SPARK-2545 Project:

[jira] [Resolved] (SPARK-2485) Usage of HiveClient not threadsafe.

2014-07-15 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-2485. --- Resolution: Fixed https://github.com/apache/spark/pull/1412 Usage of HiveClient not

[jira] [Commented] (SPARK-2154) Worker goes down.

2014-07-14 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060813#comment-14060813 ] Aaron Davidson commented on SPARK-2154: --- Created this PR to hopefully fix that:

[jira] [Created] (SPARK-2453) Compound lines in spark-shell cause compilation errors

2014-07-11 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2453: - Summary: Compound lines in spark-shell cause compilation errors Key: SPARK-2453 URL: https://issues.apache.org/jira/browse/SPARK-2453 Project: Spark Issue

[jira] [Resolved] (SPARK-2403) Spark stuck when class is not registered with Kryo

2014-07-08 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-2403. --- Resolution: Fixed Fix Version/s: 1.0.2 1.1.0 Spark stuck when

[jira] [Created] (SPARK-2412) CoalescedRDD throws exception with certain pref locs

2014-07-08 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2412: - Summary: CoalescedRDD throws exception with certain pref locs Key: SPARK-2412 URL: https://issues.apache.org/jira/browse/SPARK-2412 Project: Spark Issue

[jira] [Resolved] (SPARK-2324) SparkContext should not exit directly when spark.local.dir is a list of multiple paths and one of them has error

2014-07-03 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-2324. --- Resolution: Fixed Resolved by https://github.com/apache/spark/pull/1274 SparkContext

[jira] [Resolved] (SPARK-2349) Fix NPE in ExternalAppendOnlyMap

2014-07-03 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-2349. --- Resolution: Fixed https://github.com/apache/spark/pull/1288 Fix NPE in

[jira] [Created] (SPARK-2282) PySpark crashes if too many tasks complete quickly

2014-06-25 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2282: - Summary: PySpark crashes if too many tasks complete quickly Key: SPARK-2282 URL: https://issues.apache.org/jira/browse/SPARK-2282 Project: Spark Issue

[jira] [Resolved] (SPARK-937) Executors that exit cleanly should not have KILLED status

2014-06-15 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-937. -- Resolution: Fixed Executors that exit cleanly should not have KILLED status

[jira] [Updated] (SPARK-937) Executors that exit cleanly should not have KILLED status

2014-06-15 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-937: - Fix Version/s: 1.0.1 Executors that exit cleanly should not have KILLED status

[jira] [Created] (SPARK-2147) Master UI forgets about Executors when application exits cleanly

2014-06-14 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2147: - Summary: Master UI forgets about Executors when application exits cleanly Key: SPARK-2147 URL: https://issues.apache.org/jira/browse/SPARK-2147 Project: Spark

[jira] [Commented] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-06-12 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029498#comment-14029498 ] Aaron Davidson commented on SPARK-983: -- The idea for SizeTrackingAppendOnlyMap is that

[jira] [Created] (SPARK-2063) Creating a SchemaRDD via sql() does not correctly resolve nested types

2014-06-06 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2063: - Summary: Creating a SchemaRDD via sql() does not correctly resolve nested types Key: SPARK-2063 URL: https://issues.apache.org/jira/browse/SPARK-2063 Project:

[jira] [Created] (SPARK-2027) spark-ec2 puts Hadoop's log4j ahead of Spark's in classpath

2014-06-04 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2027: - Summary: spark-ec2 puts Hadoop's log4j ahead of Spark's in classpath Key: SPARK-2027 URL: https://issues.apache.org/jira/browse/SPARK-2027 Project: Spark

[jira] [Created] (SPARK-2028) Users of HadoopRDD cannot access the partition InputSplits

2014-06-04 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-2028: - Summary: Users of HadoopRDD cannot access the partition InputSplits Key: SPARK-2028 URL: https://issues.apache.org/jira/browse/SPARK-2028 Project: Spark

[jira] [Resolved] (SPARK-1901) Standalone worker update exector's state ahead of executor process exit

2014-05-30 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1901. --- Resolution: Fixed Standalone worker update exector's state ahead of executor process exit

[jira] [Created] (SPARK-1966) Cannot cancel tasks running locally

2014-05-29 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-1966: - Summary: Cannot cancel tasks running locally Key: SPARK-1966 URL: https://issues.apache.org/jira/browse/SPARK-1966 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010190#comment-14010190 ] Aaron Davidson commented on SPARK-983: -- Does sound reasonable. For some reason it does

[jira] [Updated] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-983: - Assignee: Madhu Siddalingaiah Support external sorting for RDD#sortByKey()

[jira] [Comment Edited] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-27 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010190#comment-14010190 ] Aaron Davidson edited comment on SPARK-983 at 5/27/14 9:54 PM:

[jira] [Commented] (SPARK-1855) Provide memory-and-local-disk RDD checkpointing

2014-05-26 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009293#comment-14009293 ] Aaron Davidson commented on SPARK-1855: --- I agree that significant improvements can

[jira] [Commented] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008554#comment-14008554 ] Aaron Davidson commented on SPARK-983: -- Historically, we have not used

[jira] [Commented] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-25 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008557#comment-14008557 ] Aaron Davidson commented on SPARK-983: -- [~pwendell] or [~matei], any opinions on

[jira] [Resolved] (SPARK-1886) workers keep dying for uncaught exception of executor id not found

2014-05-24 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1886. --- Resolution: Fixed Assignee: Zhen Peng workers keep dying for uncaught exception of

[jira] [Commented] (SPARK-983) External hashing sorting support

2014-05-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006251#comment-14006251 ] Aaron Davidson commented on SPARK-983: -- This JIRA is pretty vague. We've already

[jira] [Updated] (SPARK-983) Support external sorting for RDD#sortByKey()

2014-05-22 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-983: - Assignee: (was: Aaron Davidson) Support external sorting for RDD#sortByKey()

[jira] [Reopened] (SPARK-1689) AppClient does not respond correctly to RemoveApplication

2014-05-19 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson reopened SPARK-1689: --- The new behavior correctly informs the scheduler of the failed state, but does not exit though

[jira] [Created] (SPARK-1860) Standalone Worker cleanup should not clean up running applications by default

2014-05-16 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-1860: - Summary: Standalone Worker cleanup should not clean up running applications by default Key: SPARK-1860 URL: https://issues.apache.org/jira/browse/SPARK-1860

[jira] [Updated] (SPARK-1866) Closure cleaner does not null shadowed fields when outer scope is referenced

2014-05-16 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-1866: -- Description: Take the following example: {code} val x = 5 val instances = new

[jira] [Created] (SPARK-1865) Improve behavior of cleanup of disk state

2014-05-16 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-1865: - Summary: Improve behavior of cleanup of disk state Key: SPARK-1865 URL: https://issues.apache.org/jira/browse/SPARK-1865 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-1688) PySpark throws unhelpful exception when pyspark cannot be loaded

2014-05-15 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1688. --- Resolution: Fixed PySpark throws unhelpful exception when pyspark cannot be loaded

[jira] [Updated] (SPARK-1769) Executor loss can cause race condition in Pool

2014-05-15 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-1769: -- Description: Loss of executors (in this case due to OOMs) exposes a race condition in

[jira] [Commented] (SPARK-1767) Prefer HDFS-cached replicas when scheduling data-local tasks

2014-05-15 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997761#comment-13997761 ] Aaron Davidson commented on SPARK-1767: --- One simple workaround to this is to just

[jira] [Created] (SPARK-1771) CoarseGrainedSchedulerBackend is not resilient to Akka restarts

2014-05-15 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-1771: - Summary: CoarseGrainedSchedulerBackend is not resilient to Akka restarts Key: SPARK-1771 URL: https://issues.apache.org/jira/browse/SPARK-1771 Project: Spark

[jira] [Commented] (SPARK-1770) repartition and coalesce(shuffle=true) put objects with the same key in the same bucket

2014-05-15 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993908#comment-13993908 ] Aaron Davidson commented on SPARK-1770: --- Ah, that PR seems unrelated. repartition

[jira] [Resolved] (SPARK-1801) Open up some private APIs related to creating new RDDs for developers

2014-05-14 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1801. --- Resolution: Fixed https://github.com/apache/spark/pull/764 Open up some private APIs

[jira] [Resolved] (SPARK-1769) Executor loss can cause race condition in Pool

2014-05-14 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1769. --- Resolution: Fixed Executor loss can cause race condition in Pool

[jira] [Created] (SPARK-1772) Spark executors do not successfully die on OOM

2014-05-14 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-1772: - Summary: Spark executors do not successfully die on OOM Key: SPARK-1772 URL: https://issues.apache.org/jira/browse/SPARK-1772 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-1745) TaskContext.interrupted should probably not be a constructor argument

2014-05-14 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson resolved SPARK-1745. --- Resolution: Fixed Assignee: Andrew Or TaskContext.interrupted should probably not be

[jira] [Updated] (SPARK-1769) Executor loss can cause race condition in Pool

2014-05-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson updated SPARK-1769: -- Assignee: Andrew Or (was: Aaron Davidson) Executor loss can cause race condition in Pool

[jira] [Assigned] (SPARK-1769) Executor loss can cause race condition in Pool

2014-05-13 Thread Aaron Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Davidson reassigned SPARK-1769: - Assignee: Aaron Davidson Executor loss can cause race condition in Pool

[jira] [Created] (SPARK-1769) Executor loss can cause race condition in Pool

2014-05-13 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-1769: - Summary: Executor loss can cause race condition in Pool Key: SPARK-1769 URL: https://issues.apache.org/jira/browse/SPARK-1769 Project: Spark Issue Type:

[jira] [Created] (SPARK-1816) LiveListenerBus dies if a listener throws an exception

2014-05-12 Thread Aaron Davidson (JIRA)
Aaron Davidson created SPARK-1816: - Summary: LiveListenerBus dies if a listener throws an exception Key: SPARK-1816 URL: https://issues.apache.org/jira/browse/SPARK-1816 Project: Spark Issue

<    1   2   3   >