[jira] [Commented] (SPARK-3211) .take() is OOM-prone when there are empty partitions

[ https://issues.apache.org/jira/browse/SPARK-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124794#comment-14124794 ] Andrew Ash commented on SPARK-3211: --- This was merged into branch-1.1 and develop

[jira] [Created] (SPARK-3429) Don't include the empty string as a defaultAclUser

Andrew Ash created SPARK-3429: - Summary: Don't include the empty string as a defaultAclUser Key: SPARK-3429 URL: https://issues.apache.org/jira/browse/SPARK-3429 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2630) Input data size of CoalescedRDD is incorrect

[ https://issues.apache.org/jira/browse/SPARK-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124818#comment-14124818 ] Apache Spark commented on SPARK-2630: - User 'ash211' has created a pull request for

[jira] [Updated] (SPARK-2004) QA Automation

[ https://issues.apache.org/jira/browse/SPARK-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2004: -- Component/s: Project Infra QA Automation - Key: SPARK-2004

[jira] [Commented] (SPARK-1667) Jobs never finish successfully once bucket file missing occurred

[ https://issues.apache.org/jira/browse/SPARK-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124833#comment-14124833 ] Andrew Ash commented on SPARK-1667: --- Hi [~sarutak] it looks like you sent in a better

[jira] [Commented] (SPARK-1667) Jobs never finish successfully once bucket file missing occurred

[ https://issues.apache.org/jira/browse/SPARK-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124834#comment-14124834 ] Kousuke Saruta commented on SPARK-1667: --- [~Andrew Ash] Oh yeah, I close this ticket.

[jira] [Closed] (SPARK-1667) Jobs never finish successfully once bucket file missing occurred

[ https://issues.apache.org/jira/browse/SPARK-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta closed SPARK-1667. - Resolution: Fixed This ticket is resolved by SPARK-2670. Jobs never finish successfully once

[jira] [Commented] (SPARK-2858) Default log4j configuration no longer seems to work

[ https://issues.apache.org/jira/browse/SPARK-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124838#comment-14124838 ] Andrew Ash commented on SPARK-2858: --- Josh mentions in that ticket that the Spark EC2 AMI

[jira] [Updated] (SPARK-2553) CoGroupedRDD unnecessarily allocates a Tuple2 per dep per key

[ https://issues.apache.org/jira/browse/SPARK-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2553: -- Fix Version/s: 1.1.0 CoGroupedRDD unnecessarily allocates a Tuple2 per dep per key

[jira] [Updated] (SPARK-2574) Avoid allocating new ArrayBuffer in groupByKey's mergeCombiner

[ https://issues.apache.org/jira/browse/SPARK-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2574: -- Fix Version/s: 1.1.0 Avoid allocating new ArrayBuffer in groupByKey's mergeCombiner

[jira] [Commented] (SPARK-2048) Optimizations to CPU usage of external spilling code

[ https://issues.apache.org/jira/browse/SPARK-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124841#comment-14124841 ] Andrew Ash commented on SPARK-2048: --- All subtasks of this umbrella task have been

[jira] [Updated] (SPARK-2048) Optimizations to CPU usage of external spilling code

[ https://issues.apache.org/jira/browse/SPARK-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-2048: -- Fix Version/s: 1.1.0 Optimizations to CPU usage of external spilling code

[jira] [Commented] (SPARK-2122) Move aggregation into shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124842#comment-14124842 ] Andrew Ash commented on SPARK-2122: --- [~jerryshao] is this a dupe of SPARK-2124 ? It

[jira] [Commented] (SPARK-1956) Enable shuffle consolidation by default

[ https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124844#comment-14124844 ] Andrew Ash commented on SPARK-1956: --- [~mridulm80] there has been a significant amount of

[jira] [Updated] (SPARK-3321) Defining a class within python main script

[ https://issues.apache.org/jira/browse/SPARK-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shawn Guo updated SPARK-3321: - Priority: Minor (was: Critical) Defining a class within python main script

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124872#comment-14124872 ] Matthew Farrellee commented on SPARK-2972: -- [~roji] this was addressed for a

[jira] [Commented] (SPARK-2972) APPLICATION_COMPLETE not created in Python unless context explicitly stopped

[ https://issues.apache.org/jira/browse/SPARK-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124873#comment-14124873 ] Shay Rojansky commented on SPARK-2972: -- Thanks for answering. I guess it's a

[jira] [Updated] (SPARK-3293) yarn's web show SUCCEEDED when the driver throw a exception in yarn-client

[ https://issues.apache.org/jira/browse/SPARK-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-3293: --- Affects Version/s: 1.1.0 yarn's web show SUCCEEDED when the driver throw a exception in yarn-client

[jira] [Updated] (SPARK-3430) Introduce ValueIncrementableHashMapAccumulator to compute Histogram

[ https://issues.apache.org/jira/browse/SPARK-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Satishkumar Sheth updated SPARK-3430: --- Description: Pull request : https://github.com/apache/spark/pull/2314

[jira] [Updated] (SPARK-3430) Introduce ValueIncrementableHashMapAccumulator to compute Histogram

[ https://issues.apache.org/jira/browse/SPARK-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Satishkumar Sheth updated SPARK-3430: --- Description: Pull request : https://github.com/apache/spark/pull/2314

[jira] [Updated] (SPARK-3430) Introduce ValueIncrementableHashMapAccumulator to compute Histogram

[ https://issues.apache.org/jira/browse/SPARK-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Satishkumar Sheth updated SPARK-3430: --- Priority: Major (was: Minor) Introduce ValueIncrementableHashMapAccumulator

[jira] [Updated] (SPARK-3430) Introduce ValueIncrementableHashMapAccumulator to compute Histogram and other statistical metrics

[ https://issues.apache.org/jira/browse/SPARK-3430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Satishkumar Sheth updated SPARK-3430: --- Summary: Introduce ValueIncrementableHashMapAccumulator to compute Histogram

[jira] [Commented] (SPARK-1087) Separate file for traceback and callsite related functions

[ https://issues.apache.org/jira/browse/SPARK-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124999#comment-14124999 ] Jyotiska NK commented on SPARK-1087: We initially thought this would be a good feature

[jira] [Comment Edited] (SPARK-1087) Separate file for traceback and callsite related functions

[ https://issues.apache.org/jira/browse/SPARK-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124999#comment-14124999 ] Jyotiska NK edited comment on SPARK-1087 at 9/7/14 6:32 PM: We

[jira] [Comment Edited] (SPARK-1087) Separate file for traceback and callsite related functions

[ https://issues.apache.org/jira/browse/SPARK-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124999#comment-14124999 ] Jyotiska NK edited comment on SPARK-1087 at 9/7/14 6:33 PM: We

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125008#comment-14125008 ] Brock Noland commented on SPARK-3174: - Thank you all for your work on this issue! I am

[jira] [Commented] (SPARK-1956) Enable shuffle consolidation by default

[ https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125027#comment-14125027 ] Mridul Muralidharan commented on SPARK-1956: The recent changes to

[jira] [Updated] (SPARK-2232) Fix Jenkins tests in Maven

[ https://issues.apache.org/jira/browse/SPARK-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2232: -- Priority: Critical (was: Major) Fix Jenkins tests in Maven --

[jira] [Updated] (SPARK-2232) Fix Jenkins tests in Maven

[ https://issues.apache.org/jira/browse/SPARK-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2232: -- Priority: Blocker (was: Critical) Fix Jenkins tests in Maven --

[jira] [Updated] (SPARK-2232) Fix Jenkins tests in Maven

[ https://issues.apache.org/jira/browse/SPARK-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2232: -- Description: It appears Maven tests are failing under the newer Hadoop configurations. We need to go

[jira] [Created] (SPARK-3431) Parallelize execution of tests

Nicholas Chammas created SPARK-3431: --- Summary: Parallelize execution of tests Key: SPARK-3431 URL: https://issues.apache.org/jira/browse/SPARK-3431 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-3432) Fix logging of unit test execution time

Nicholas Chammas created SPARK-3432: --- Summary: Fix logging of unit test execution time Key: SPARK-3432 URL: https://issues.apache.org/jira/browse/SPARK-3432 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-3394) TakeOrdered crashes when limit is 0

[ https://issues.apache.org/jira/browse/SPARK-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia resolved SPARK-3394. -- Resolution: Fixed Fix Version/s: 1.0.3 1.2.0 1.1.1

[jira] [Updated] (SPARK-3394) TakeOrdered crashes when limit is 0

[ https://issues.apache.org/jira/browse/SPARK-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-3394: - Component/s: Spark Core TakeOrdered crashes when limit is 0 ---

[jira] [Updated] (SPARK-3263) PR #720 broke GraphGenerator.logNormal

[ https://issues.apache.org/jira/browse/SPARK-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3263: --- Fix Version/s: (was: 1.3.0) 1.2.0 PR #720 broke

[jira] [Commented] (SPARK-3360) Add RowMatrix.multiply(Vector)

[ https://issues.apache.org/jira/browse/SPARK-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125129#comment-14125129 ] Yu Ishikawa commented on SPARK-3360: Hi Sandy, I'm interested in this issue. It seems

[jira] [Resolved] (SPARK-3408) Limit operator doesn't work with sort based shuffle

[ https://issues.apache.org/jira/browse/SPARK-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3408. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Limit operator doesn't work

[jira] [Created] (SPARK-3433) Mima false-positives with @DeveloperAPI and @Experimental annotations

Josh Rosen created SPARK-3433: - Summary: Mima false-positives with @DeveloperAPI and @Experimental annotations Key: SPARK-3433 URL: https://issues.apache.org/jira/browse/SPARK-3433 Project: Spark

[jira] [Resolved] (SPARK-3415) Using sys.stderr in pyspark results in error

[ https://issues.apache.org/jira/browse/SPARK-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3415. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2287

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125172#comment-14125172 ] Patrick Wendell commented on SPARK-3174: Hey Sandy - thanks for posting the design

[jira] [Resolved] (SPARK-3280) Made sort-based shuffle the default implementation

[ https://issues.apache.org/jira/browse/SPARK-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-3280. Resolution: Fixed Fix Version/s: 1.2.0 Made sort-based shuffle the default implementation

[jira] [Resolved] (SPARK-938) OpenStack Swift Storage Support

[ https://issues.apache.org/jira/browse/SPARK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-938. --- Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request 2298

[jira] [Updated] (SPARK-938) OpenStack Swift Storage Support

[ https://issues.apache.org/jira/browse/SPARK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-938: -- Assignee: Gil Vernik OpenStack Swift Storage Support ---

[jira] [Commented] (SPARK-938) OpenStack Swift Storage Support

[ https://issues.apache.org/jira/browse/SPARK-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125195#comment-14125195 ] Patrick Wendell commented on SPARK-938: --- This was fixed by [~gvernik] with [~rxin]

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125235#comment-14125235 ] Sandy Ryza commented on SPARK-3174: --- To be clear, by YARN shuffle you mean the MR2

[jira] [Updated] (SPARK-2425) Standalone Master is too aggressive in removing Applications

[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2425: --- Priority: Critical (was: Major) Standalone Master is too aggressive in removing

[jira] [Updated] (SPARK-2425) Standalone Master is too aggressive in removing Applications

[ https://issues.apache.org/jira/browse/SPARK-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2425: --- Target Version/s: 1.2.0 (was: 1.0.3) Standalone Master is too aggressive in removing