[jira] [Commented] (SPARK-3622) Provide a custom transformation that can output multiple RDDs

2014-09-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142865#comment-14142865 ] Patrick Wendell commented on SPARK-3622: Do you mind clarifying a little bit how h

[jira] [Comment Edited] (SPARK-3622) Provide a custom transformation that can output multiple RDDs

2014-09-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142865#comment-14142865 ] Patrick Wendell edited comment on SPARK-3622 at 9/22/14 3:24 AM: ---

[jira] [Updated] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-09-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3633: --- Summary: Fetches failure observed after SPARK-2711 (was: PR 1707/commit #4fde28c is problemat

[jira] [Commented] (SPARK-3622) Provide a custom transformation that can output multiple RDDs

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142982#comment-14142982 ] Patrick Wendell commented on SPARK-3622: In Spark most RDD operations are lazy, so

[jira] [Comment Edited] (SPARK-3622) Provide a custom transformation that can output multiple RDDs

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142982#comment-14142982 ] Patrick Wendell edited comment on SPARK-3622 at 9/22/14 7:35 AM: ---

[jira] [Comment Edited] (SPARK-3622) Provide a custom transformation that can output multiple RDDs

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142982#comment-14142982 ] Patrick Wendell edited comment on SPARK-3622 at 9/22/14 7:35 AM: ---

[jira] [Created] (SPARK-3648) Provide a script for fetching remote PR's for review

2014-09-22 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3648: -- Summary: Provide a script for fetching remote PR's for review Key: SPARK-3648 URL: https://issues.apache.org/jira/browse/SPARK-3648 Project: Spark Issue

[jira] [Updated] (SPARK-3648) Provide a script for fetching remote PR's for review

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3648: --- Issue Type: New Feature (was: Bug) > Provide a script for fetching remote PR's for review > -

[jira] [Updated] (SPARK-1720) use LD_LIBRARY_PATH instead of -Djava.library.path

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1720: --- Priority: Critical (was: Major) Target Version/s: 1.2.0 > use LD_LIBRARY_PATH ins

[jira] [Commented] (SPARK-1720) use LD_LIBRARY_PATH instead of -Djava.library.path

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143930#comment-14143930 ] Patrick Wendell commented on SPARK-1720: Another user reported this issue, so let'

[jira] [Updated] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1860: --- Fix Version/s: (was: 1.2.0) > Standalone Worker cleanup should not clean up running execut

[jira] [Updated] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1860: --- Priority: Blocker (was: Critical) > Standalone Worker cleanup should not clean up running exe

[jira] [Updated] (SPARK-1860) Standalone Worker cleanup should not clean up running executors

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1860: --- Target Version/s: 1.2.0 > Standalone Worker cleanup should not clean up running executors > --

[jira] [Updated] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3032: --- Priority: Critical (was: Major) > Potential bug when running sort-based shuffle with sorting

[jira] [Updated] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

2014-09-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3032: --- Target Version/s: 1.2.0 > Potential bug when running sort-based shuffle with sorting using Tim

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-09-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144490#comment-14144490 ] Patrick Wendell commented on SPARK-3633: [~nravi] if you are trying to debug this,

[jira] [Resolved] (SPARK-3647) Shaded Guava patch causes access issues with package private classes

2014-09-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3647. Resolution: Fixed Fixed by Marcelo in this patch: https://github.com/apache/spark/pull/2496

[jira] [Resolved] (SPARK-3612) Executor shouldn't quit if heartbeat message fails to reach the driver

2014-09-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3612. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Assignee: San

[jira] [Updated] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

2014-09-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3032: --- Priority: Blocker (was: Critical) > Potential bug when running sort-based shuffle with sortin

[jira] [Resolved] (SPARK-3659) Set EC2 version to 1.1.0 in master branch

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3659. Resolution: Fixed Fix Version/s: 1.2.0 https://github.com/apache/spark/pull/2510 > S

[jira] [Updated] (SPARK-2691) Allow Spark on Mesos to be launched with Docker

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2691: --- Assignee: Timothy Hunter > Allow Spark on Mesos to be launched with Docker > -

[jira] [Updated] (SPARK-2691) Allow Spark on Mesos to be launched with Docker

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2691: --- Assignee: Timothy Chen (was: Tim Chen) > Allow Spark on Mesos to be launched with Docker > --

[jira] [Updated] (SPARK-2691) Allow Spark on Mesos to be launched with Docker

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2691: --- Assignee: Tim Chen (was: Timothy Hunter) > Allow Spark on Mesos to be launched with Docker >

[jira] [Resolved] (SPARK-3604) unbounded recursion in getNumPartitions triggers stack overflow for large UnionRDD

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3604. Resolution: Not a Problem > unbounded recursion in getNumPartitions triggers stack overflow

[jira] [Updated] (SPARK-3681) Failed to serialized ArrayType or MapType after accessing them in Python

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3681: --- Component/s: PySpark > Failed to serialized ArrayType or MapType after accessing them in Pyth

[jira] [Updated] (SPARK-3663) Document SPARK_LOG_DIR and SPARK_PID_DIR

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3663: --- Component/s: Documentation > Document SPARK_LOG_DIR and SPARK_PID_DIR > --

[jira] [Updated] (SPARK-3610) History server log name should not be based on user input

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3610: --- Component/s: Spark Core > History server log name should not be based on user input >

[jira] [Resolved] (SPARK-3615) Kafka test should not hard code Zookeeper port

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3615. Resolution: Fixed https://github.com/apache/spark/pull/2483 > Kafka test should not hard co

[jira] [Created] (SPARK-3686) flume.SparkSinkSuite.Success is flaky

2014-09-24 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3686: -- Summary: flume.SparkSinkSuite.Success is flaky Key: SPARK-3686 URL: https://issues.apache.org/jira/browse/SPARK-3686 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-546) Support full outer join and multiple join in a single shuffle

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-546. --- Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Aaron Staple Fixed by: https:/

[jira] [Resolved] (SPARK-2778) Add unit tests for Yarn integration

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2778. Resolution: Fixed Fix Version/s: 1.2.0 Fixed by: https://github.com/apache/spark/pull

[jira] [Commented] (SPARK-3687) Spark hang while processing more than 100 sequence files

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147465#comment-14147465 ] Patrick Wendell commented on SPARK-3687: Can you perform a jstack on the executor

[jira] [Resolved] (SPARK-3576) Provide script for creating the Spark AMI from scratch

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3576. Resolution: Fixed This was fixed in spark-ec2 itself > Provide script for creating the Spar

[jira] [Updated] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3288: --- Assignee: (was: Andrew Or) > All fields in TaskMetrics should be private and use getters/s

[jira] [Updated] (SPARK-3288) All fields in TaskMetrics should be private and use getters/setters

2014-09-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3288: --- Labels: starter (was: ) > All fields in TaskMetrics should be private and use getters/setters

[jira] [Created] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-09-25 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3694: -- Summary: Allow printing object graph of tasks/RDD's with a debug flag Key: SPARK-3694 URL: https://issues.apache.org/jira/browse/SPARK-3694 Project: Spark

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Description: This would be useful for debugging extra references inside of RDD's Here is an e

[jira] [Resolved] (SPARK-3584) sbin/slaves doesn't work when we use password authentication for SSH

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3584. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Kousuke Saruta

[jira] [Resolved] (SPARK-3686) flume.SparkSinkSuite.Success is flaky

2014-09-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3686. Resolution: Fixed Resolved by: https://github.com/apache/spark/pull/2531 > flume.SparkSinkS

[jira] [Updated] (SPARK-3695) Enable to show host and port in block fetch failure

2014-09-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3695: --- Assignee: Adrian Wang (was: Patrick Wendell) > Enable to show host and port in block fetch fa

[jira] [Resolved] (SPARK-3695) Enable to show host and port in block fetch failure

2014-09-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3695. Resolution: Fixed Fix Version/s: 1.2.0 > Enable to show host and port in block fetch

[jira] [Resolved] (SPARK-2655) Change the default logging level to WARN

2014-09-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2655. Resolution: Won't Fix > Change the default logging level to WARN > -

[jira] [Created] (SPARK-3709) BroadcastSuite.Unpersisting rg.apache.spark.broadcast.BroadcastSuite.Unpersisting TorrentBroadcast is flaky

2014-09-27 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3709: -- Summary: BroadcastSuite.Unpersisting rg.apache.spark.broadcast.BroadcastSuite.Unpersisting TorrentBroadcast is flaky Key: SPARK-3709 URL: https://issues.apache.org/jira/brows

[jira] [Created] (SPARK-3710) YARN integration test is flaky

2014-09-27 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3710: -- Summary: YARN integration test is flaky Key: SPARK-3710 URL: https://issues.apache.org/jira/browse/SPARK-3710 Project: Spark Issue Type: Bug Co

[jira] [Updated] (SPARK-3710) YARN integration test is flaky

2014-09-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3710: --- Description: This has been regularly failing the master build: Example failure: https://ampl

[jira] [Commented] (SPARK-3685) Spark's local dir scheme is not configurable

2014-09-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151350#comment-14151350 ] Patrick Wendell commented on SPARK-3685: [~andrewor14] changing the use of local s

[jira] [Updated] (SPARK-2626) Stop SparkContext in all examples

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2626: --- Labels: starter (was: ) > Stop SparkContext in all examples > ---

[jira] [Commented] (SPARK-2805) update akka to version 2.3

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151976#comment-14151976 ] Patrick Wendell commented on SPARK-2805: I'm working on publishing akka today. >

[jira] [Commented] (SPARK-3479) Have Jenkins show which tests failed in his GitHub messages

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151980#comment-14151980 ] Patrick Wendell commented on SPARK-3479: I think this is more than "minor" - it wo

[jira] [Updated] (SPARK-3479) Have Jenkins show which tests failed in his GitHub messages

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3479: --- Priority: Major (was: Minor) > Have Jenkins show which tests failed in his GitHub messages >

[jira] [Updated] (SPARK-3479) Have Jenkins show which tests failed in his GitHub messages

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3479: --- Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-2230) > Have Jenkins

[jira] [Resolved] (SPARK-2230) Improvements to Jenkins QA Harness

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2230. Resolution: Fixed This was tracking an earlier initiative to clean up this harness. Since we

[jira] [Commented] (SPARK-2331) SparkContext.emptyRDD has wrong return type

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152018#comment-14152018 ] Patrick Wendell commented on SPARK-2331: Yeah we could have made this a wider type

[jira] [Resolved] (SPARK-2331) SparkContext.emptyRDD has wrong return type

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2331. Resolution: Won't Fix > SparkContext.emptyRDD has wrong return type > --

[jira] [Comment Edited] (SPARK-2331) SparkContext.emptyRDD has wrong return type

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152018#comment-14152018 ] Patrick Wendell edited comment on SPARK-2331 at 9/29/14 6:34 PM: ---

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Labels: starter (was: ) > Allow printing object graph of tasks/RDD's with a debug flag >

[jira] [Updated] (SPARK-3736) Workers should reconnect to Master if disconnected

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3736: --- Priority: Critical (was: Major) Target Version/s: 1.2.0 > Workers should reconnec

[jira] [Updated] (SPARK-2548) JavaRecoverableWordCount is missing

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2548: --- Labels: starter (was: ) > JavaRecoverableWordCount is missing > -

[jira] [Updated] (SPARK-3504) KMeans optimization: track distances and unmoved cluster centers across iterations

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3504: --- Summary: KMeans optimization: track distances and unmoved cluster centers across iterations (

[jira] [Commented] (SPARK-3504) KMeans optimization: track distances and unmoved cluster centers across iterations

2014-09-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152730#comment-14152730 ] Patrick Wendell commented on SPARK-3504: I just updated the title to make it more

[jira] [Reopened] (SPARK-3007) Add "Dynamic Partition" support to Spark Sql hive

2014-09-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3007: This was reverted based on causing large numbers of test failures. > Add "Dynamic Partition" su

[jira] [Reopened] (SPARK-2778) Add unit tests for Yarn integration

2014-09-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-2778: This has been reverted due to failing tests. > Add unit tests for Yarn integration > --

[jira] [Updated] (SPARK-2778) Add unit tests for Yarn integration

2014-09-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2778: --- Attachment: yarn-logs.txt I'm attaching logs from the bad test. > Add unit tests for Yarn int

[jira] [Updated] (SPARK-2778) Add unit tests for Yarn integration

2014-09-30 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2778: --- Description: It would be nice to add some Yarn integration tests to the unit tests in Spark;

[jira] [Created] (SPARK-3744) FlumeStreamSuite will fail during port contention

2014-09-30 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3744: -- Summary: FlumeStreamSuite will fail during port contention Key: SPARK-3744 URL: https://issues.apache.org/jira/browse/SPARK-3744 Project: Spark Issue Typ

[jira] [Resolved] (SPARK-3757) mvn clean doesn't delete some files

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3757. Resolution: Fixed Fix Version/s: 1.2.0 Resolved by: https://github.com/apache/spark/p

[jira] [Resolved] (SPARK-3756) Include possible MultiException when detecting port collisions

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3756. Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Assignee: wan

[jira] [Updated] (SPARK-3756) Include possible MultiException when detecting port collisions

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3756: --- Summary: Include possible MultiException when detecting port collisions (was: check exception

[jira] [Reopened] (SPARK-3755) Do not bind port 1 - 1024 to server in spark

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3755: Original fix was broken so I'm re-opening it. > Do not bind port 1 - 1024 to server in spark >

[jira] [Resolved] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3292. Resolution: Won't Fix Seems like this is a necessary feature of the current design and can b

[jira] [Commented] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155829#comment-14155829 ] Patrick Wendell commented on SPARK-3761: Is this an sbt bug rather than a spark on

[jira] [Comment Edited] (SPARK-3761) Class anonfun$1 not found exception / sbt 13.x / Scala 2.10.4

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155829#comment-14155829 ] Patrick Wendell edited comment on SPARK-3761 at 10/2/14 12:03 AM: --

[jira] [Commented] (SPARK-3759) SparkSubmitDriverBootstrapper should return exit code of driver process

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155833#comment-14155833 ] Patrick Wendell commented on SPARK-3759: Thanks for reporting this Eric - do you w

[jira] [Resolved] (SPARK-3730) Any one else having building spark recently

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3730. Resolution: Invalid For this type of question checkout the Spark dev list. To answer your sp

[jira] [Commented] (SPARK-3731) RDD caching stops working in pyspark after some time

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155844#comment-14155844 ] Patrick Wendell commented on SPARK-3731: Thanks for reporting this - can you repro

[jira] [Commented] (SPARK-3573) Dataset

2014-10-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156129#comment-14156129 ] Patrick Wendell commented on SPARK-3573: I think people are hung up on the term SQ

[jira] [Resolved] (SPARK-1767) Prefer HDFS-cached replicas when scheduling data-local tasks

2014-10-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1767. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Colin Patrick McCabe Fixed

[jira] [Resolved] (SPARK-3007) Add "Dynamic Partition" support to Spark Sql hive

2014-10-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3007. Resolution: Fixed Okay, this was merged again: https://github.com/apache/spark/pull/2616 >

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3174: --- Summary: Provide elastic scaling within a Spark application (was: Under YARN, add and remove

[jira] [Commented] (SPARK-3174) Under YARN, add and remove executors based on load

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159250#comment-14159250 ] Patrick Wendell commented on SPARK-3174: Hey all - I'm going to restructure this J

[jira] [Resolved] (SPARK-3460) Under YARN, discard executors that have been idle

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3460. Resolution: Invalid This is going to be covered by a re-organization of this JIRA. > Under

[jira] [Updated] (SPARK-3460) Under YARN, discard executors that have been idle

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3460: --- Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-3174) > Under YARN, discard e

[jira] [Updated] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3464: --- Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-3174) > Graceful decommission

[jira] [Resolved] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3464. Resolution: Fixed > Graceful decommission of executors > --

[jira] [Created] (SPARK-3795) Provide scheduler hooks for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3795: -- Summary: Provide scheduler hooks for adding and removing executors Key: SPARK-3795 URL: https://issues.apache.org/jira/browse/SPARK-3795 Project: Spark I

[jira] [Created] (SPARK-3796) Provide shuffle service for external block storage

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3796: -- Summary: Provide shuffle service for external block storage Key: SPARK-3796 URL: https://issues.apache.org/jira/browse/SPARK-3796 Project: Spark Issue Ty

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Summary: Add scheduler hooks/heuristics for adding and removing executors (was: Provide sched

[jira] [Updated] (SPARK-3796) Create shuffle service for external block storage

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3796: --- Summary: Create shuffle service for external block storage (was: Provide shuffle service for

[jira] [Created] (SPARK-3797) Integrate shuffle service in YARN's pluggable shuffle

2014-10-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3797: -- Summary: Integrate shuffle service in YARN's pluggable shuffle Key: SPARK-3797 URL: https://issues.apache.org/jira/browse/SPARK-3797 Project: Spark Issue

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Description: To support dynamic scaling of a Spark application, Spark's scheduler will need to

[jira] [Updated] (SPARK-3795) Add scheduler hooks/heuristics for adding and removing executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3795: --- Description: To support dynamic scaling of a Spark application, Spark's scheduler will need to

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3174: --- Component/s: (was: YARN) > Provide elastic scaling within a Spark application > --

[jira] [Updated] (SPARK-3794) Building spark core fails with specific hadoop version

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3794: --- Fix Version/s: (was: 1.2.0) > Building spark core fails with specific hadoop version > ---

[jira] [Commented] (SPARK-3794) Building spark core fails with specific hadoop version

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159330#comment-14159330 ] Patrick Wendell commented on SPARK-3794: Are you trying to build from within the c

[jira] [Reopened] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3464: > Graceful decommission of executors > -- > > Ke

[jira] [Resolved] (SPARK-3464) Graceful decommission of executors

2014-10-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3464. Resolution: Invalid This is going to be subsumed by a separate JIRA. > Graceful decommissio

[jira] [Commented] (SPARK-3805) Enable Standalone worker cleanup by default

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159837#comment-14159837 ] Patrick Wendell commented on SPARK-3805: I commented on the PR - but because this

[jira] [Commented] (SPARK-3561) Native Hadoop/YARN integration for batch/ETL workloads.

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159838#comment-14159838 ] Patrick Wendell commented on SPARK-3561: [~ozhurakousky] do you have a timeline fo

[jira] [Updated] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3561: --- Summary: Allow for pluggable execution contexts in Spark (was: Native Hadoop/YARN integration

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159840#comment-14159840 ] Patrick Wendell commented on SPARK-3561: I also changed the title here that reflec

<    1   2   3   4   5   6   7   8   9   10   >