[jira] [Issue Comment Deleted] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions

2014-10-23 Thread jeanlyn (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-3967: --- Comment: was deleted (was: dsa dsa) Spark applications fail in yarn-cluster mode when the

[jira] [Commented] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions

2014-10-23 Thread jeanlyn (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181057#comment-14181057 ] jeanlyn commented on SPARK-3967: dsa dsa Spark applications fail in yarn-cluster

[jira] [Created] (SPARK-4058) Log file name is hard coded even though there is a variable '$LOG_FILE '

2014-10-23 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-4058: - Summary: Log file name is hard coded even though there is a variable '$LOG_FILE ' Key: SPARK-4058 URL: https://issues.apache.org/jira/browse/SPARK-4058 Project:

[jira] [Commented] (SPARK-4058) Log file name is hard coded even though there is a variable '$LOG_FILE '

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181060#comment-14181060 ] Apache Spark commented on SPARK-4058: - User 'sarutak' has created a pull request for

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181066#comment-14181066 ] Patrick Wendell commented on SPARK-3655: Hey [~koertkuipers] - i'm not an expert

[jira] [Created] (SPARK-4059) spark-master/spark-worker may use SPARK_MASTER_IP/STANDALONE_SPARK_MASTER_HOST

2014-10-23 Thread Guo Ruijing (JIRA)
Guo Ruijing created SPARK-4059: -- Summary: spark-master/spark-worker may use SPARK_MASTER_IP/STANDALONE_SPARK_MASTER_HOST Key: SPARK-4059 URL: https://issues.apache.org/jira/browse/SPARK-4059 Project:

[jira] [Updated] (SPARK-4020) Failed executor not properly removed if it has not run tasks

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4020: --- Component/s: Spark Core Failed executor not properly removed if it has not run tasks

[jira] [Commented] (SPARK-3254) Streaming K-Means

2014-10-23 Thread Jeremy Freeman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181080#comment-14181080 ] Jeremy Freeman commented on SPARK-3254: --- I'd like it to! I've got a PR ready to

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

2014-10-23 Thread Gen TANG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181120#comment-14181120 ] Gen TANG commented on SPARK-1977: - [~sinisa_lyh] Sorry to bother you. According to

[jira] [Commented] (SPARK-2429) Hierarchical Implementation of KMeans

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181167#comment-14181167 ] Apache Spark commented on SPARK-2429: - User 'yu-iskw' has created a pull request for

[jira] [Commented] (SPARK-2429) Hierarchical Implementation of KMeans

2014-10-23 Thread Yu Ishikawa (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181169#comment-14181169 ] Yu Ishikawa commented on SPARK-2429: Hi [~rnowling], I sent the PR. Could you review

[jira] [Commented] (SPARK-2429) Hierarchical Implementation of KMeans

2014-10-23 Thread RJ Nowling (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181183#comment-14181183 ] RJ Nowling commented on SPARK-2429: --- I added a couple comments to the PR. I would say

[jira] [Commented] (SPARK-3928) Support wildcard matches on Parquet files

2014-10-23 Thread Marius Soutier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181185#comment-14181185 ] Marius Soutier commented on SPARK-3928: --- This would be more than nice. Currently,

[jira] [Created] (SPARK-4060) MLlib, exposing special rdd functions to the public

2014-10-23 Thread Niklas Wilcke (JIRA)
Niklas Wilcke created SPARK-4060: Summary: MLlib, exposing special rdd functions to the public Key: SPARK-4060 URL: https://issues.apache.org/jira/browse/SPARK-4060 Project: Spark Issue

[jira] [Commented] (SPARK-3954) Optimization to FileInputDStream

2014-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181190#comment-14181190 ] 宿荣全 commented on SPARK-3954: Does someone take notice of this PR? Optimization to

[jira] [Commented] (SPARK-4060) MLlib, exposing special rdd functions to the public

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181193#comment-14181193 ] Apache Spark commented on SPARK-4060: - User 'numbnut' has created a pull request for

[jira] [Created] (SPARK-4061) We cannot use EOL character in the operand of LIKE predicate.

2014-10-23 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-4061: - Summary: We cannot use EOL character in the operand of LIKE predicate. Key: SPARK-4061 URL: https://issues.apache.org/jira/browse/SPARK-4061 Project: Spark

[jira] [Commented] (SPARK-4061) We cannot use EOL character in the operand of LIKE predicate.

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181196#comment-14181196 ] Apache Spark commented on SPARK-4061: - User 'sarutak' has created a pull request for

[jira] [Commented] (SPARK-1473) Feature selection for high dimensional datasets

2014-10-23 Thread Gavin Brown (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181238#comment-14181238 ] Gavin Brown commented on SPARK-1473: Hello, I am the first author of the paper being

[jira] [Comment Edited] (SPARK-1473) Feature selection for high dimensional datasets

2014-10-23 Thread Gavin Brown (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181238#comment-14181238 ] Gavin Brown edited comment on SPARK-1473 at 10/23/14 11:20 AM:

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-23 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181248#comment-14181248 ] koert kuipers commented on SPARK-3655: -- hey patrick. i was looking into modifying the

[jira] [Commented] (SPARK-2090) spark-shell input text entry not showing on REPL

2014-10-23 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181273#comment-14181273 ] sam commented on SPARK-2090: I've experienced the same problem when I login to a cluster where

[jira] [Commented] (SPARK-2652) Turning default configurations for PySpark

2014-10-23 Thread Gen TANG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181282#comment-14181282 ] Gen TANG commented on SPARK-2652: - If we set {code} spark.serializer,

[jira] [Comment Edited] (SPARK-2652) Turning default configurations for PySpark

2014-10-23 Thread Gen TANG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181282#comment-14181282 ] Gen TANG edited comment on SPARK-2652 at 10/23/14 12:29 PM: If

[jira] [Commented] (SPARK-1977) mutable.BitSet in ALS not serializable with KryoSerializer

2014-10-23 Thread Gen TANG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181287#comment-14181287 ] Gen TANG commented on SPARK-1977: - In fact, the problem about transformation between

[jira] [Commented] (SPARK-732) Recomputation of RDDs may result in duplicated accumulator updates

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181318#comment-14181318 ] Apache Spark commented on SPARK-732: User 'CodingCat' has created a pull request for

[jira] [Commented] (SPARK-3924) Upgrade to Akka version 2.3.6

2014-10-23 Thread Helena Edelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181319#comment-14181319 ] Helena Edelson commented on SPARK-3924: --- I met with Matei at Strata about this.

[jira] [Created] (SPARK-4062) Improve KafkaReceiver to prevent data loss

2014-10-23 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-4062: -- Summary: Improve KafkaReceiver to prevent data loss Key: SPARK-4062 URL: https://issues.apache.org/jira/browse/SPARK-4062 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

2014-10-23 Thread Helena Edelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Helena Edelson resolved SPARK-2593. --- Resolution: Won't Fix As a user, I want to be able to use the latest version of Akka with

[jira] [Updated] (SPARK-4062) Improve KafkaReceiver to prevent data loss

2014-10-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-4062: --- Attachment: RefactoredKafkaReceiver.pdf Improve KafkaReceiver to prevent data loss

[jira] [Created] (SPARK-4063) Add the ability to send messages to Kafka in the stream

2014-10-23 Thread Helena Edelson (JIRA)
Helena Edelson created SPARK-4063: - Summary: Add the ability to send messages to Kafka in the stream Key: SPARK-4063 URL: https://issues.apache.org/jira/browse/SPARK-4063 Project: Spark

[jira] [Created] (SPARK-4064) In the case of creating A lot of uge broadcast variable,spark hangs.

2014-10-23 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-4064: -- Summary: In the case of creating A lot of uge broadcast variable,spark hangs. Key: SPARK-4064 URL: https://issues.apache.org/jira/browse/SPARK-4064 Project: Spark

[jira] [Updated] (SPARK-4064) In the case of creating A lot of uge broadcast variable,spark hangs.

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Attachment: executor.log In the case of creating A lot of uge broadcast variable,spark hangs.

[jira] [Updated] (SPARK-4064) In the case of creating A lot of uge broadcast variable,spark hangs.

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Attachment: screenshot.png In the case of creating A lot of uge broadcast variable,spark hangs.

[jira] [Updated] (SPARK-4064) In the case of creating A lot of uge broadcast variable,spark hangs.

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Fix Version/s: 1.2.0 In the case of creating A lot of uge broadcast variable,spark hangs.

[jira] [Updated] (SPARK-4064) In the case of creating A lot of huge broadcast variable,spark hangs.

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Summary: In the case of creating A lot of huge broadcast variable,spark hangs. (was: In the case of

[jira] [Updated] (SPARK-4064) In the case of creating a lot of huge broadcast variable,spark hangs.

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Summary: In the case of creating a lot of huge broadcast variable,spark hangs. (was: In the case of

[jira] [Updated] (SPARK-4064) In the case of creating a lot of huge broadcast variable,spark hangs.

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Attachment: jstack.txt In the case of creating a lot of huge broadcast variable,spark hangs.

[jira] [Commented] (SPARK-3359) `sbt/sbt unidoc` doesn't work with Java 8

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181494#comment-14181494 ] Apache Spark commented on SPARK-3359: - User 'srowen' has created a pull request for

[jira] [Commented] (SPARK-3359) `sbt/sbt unidoc` doesn't work with Java 8

2014-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181503#comment-14181503 ] Sean Owen commented on SPARK-3359: -- I inquired about these with the plugin project:

[jira] [Commented] (SPARK-4063) Add the ability to send messages to Kafka in the stream

2014-10-23 Thread Helena Edelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181520#comment-14181520 ] Helena Edelson commented on SPARK-4063: --- I have this started in a WIP branch Add

[jira] [Commented] (SPARK-4022) Replace colt dependency (LGPL) with commons-math

2014-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181524#comment-14181524 ] Sean Owen commented on SPARK-4022: -- I have begun work on this. You can see the base

[jira] [Commented] (SPARK-4063) Add the ability to send messages to Kafka in the stream

2014-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181526#comment-14181526 ] Sean Owen commented on SPARK-4063: -- Is this like a streaming operation that saves an RDD

[jira] [Resolved] (SPARK-4055) Inconsistent spelling 'MLlib' and 'MLLib'

2014-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4055. -- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2903

[jira] [Updated] (SPARK-4055) Inconsistent spelling 'MLlib' and 'MLLib'

2014-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-4055: - Assignee: Kousuke Saruta Inconsistent spelling 'MLlib' and 'MLLib'

[jira] [Updated] (SPARK-4019) Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or

[jira] [Created] (SPARK-4065) pyspark will not use ipython on Windows

2014-10-23 Thread Michael Griffiths (JIRA)
Michael Griffiths created SPARK-4065: Summary: pyspark will not use ipython on Windows Key: SPARK-4065 URL: https://issues.apache.org/jira/browse/SPARK-4065 Project: Spark Issue Type:

[jira] [Commented] (SPARK-4065) pyspark will not use ipython on Windows

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181637#comment-14181637 ] Apache Spark commented on SPARK-4065: - User 'msjgriffiths' has created a pull request

[jira] [Updated] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4006: - Fix Version/s: 1.2.0 Spark Driver crashes whenever an Executor is registered twice

[jira] [Updated] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4006: - Assignee: Tal Sliwowicz Spark Driver crashes whenever an Executor is registered twice

[jira] [Commented] (SPARK-4056) Upgrade snappy-java to 1.1.1.4

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181757#comment-14181757 ] Apache Spark commented on SPARK-4056: - User 'JoshRosen' has created a pull request for

[jira] [Commented] (SPARK-4022) Replace colt dependency (LGPL) with commons-math

2014-10-23 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181777#comment-14181777 ] Joseph K. Bradley commented on SPARK-4022: -- Hi Sean, Thanks for picking this up!

[jira] [Comment Edited] (SPARK-4022) Replace colt dependency (LGPL) with commons-math

2014-10-23 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181777#comment-14181777 ] Joseph K. Bradley edited comment on SPARK-4022 at 10/23/14 7:00 PM:

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181792#comment-14181792 ] Patrick Wendell commented on SPARK-3655: Yeah so to be clear here is what I meant:

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-23 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181807#comment-14181807 ] koert kuipers commented on SPARK-3655: -- yes, that makes sense. i am working right

[jira] [Updated] (SPARK-4050) Caching of temporary tables with projects fail when the final query projects fewer columns

2014-10-23 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4050: Summary: Caching of temporary tables with projects fail when the final query projects fewer

[jira] [Commented] (SPARK-4022) Replace colt dependency (LGPL) with commons-math

2014-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181886#comment-14181886 ] Xiangrui Meng commented on SPARK-4022: -- Hi [~srowen], ChiSqTest: This is an

[jira] [Commented] (SPARK-4050) Caching of temporary tables with projects fail when the final query projects fewer columns

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181890#comment-14181890 ] Apache Spark commented on SPARK-4050: - User 'marmbrus' has created a pull request for

[jira] [Updated] (SPARK-4066) Make whether maven builds fails on scalastyle violation configurable

2014-10-23 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated SPARK-4066: -- Attachment: spark-4066-v1.txt With attached patch, developer can specify the following on command line:

[jira] [Created] (SPARK-4066) Make whether maven builds fails on scalastyle violation configurable

2014-10-23 Thread Ted Yu (JIRA)
Ted Yu created SPARK-4066: - Summary: Make whether maven builds fails on scalastyle violation configurable Key: SPARK-4066 URL: https://issues.apache.org/jira/browse/SPARK-4066 Project: Spark Issue

[jira] [Created] (SPARK-4067) refactor ExecutorUncaughtExceptionHandler as a general one as it is used like this

2014-10-23 Thread Nan Zhu (JIRA)
Nan Zhu created SPARK-4067: -- Summary: refactor ExecutorUncaughtExceptionHandler as a general one as it is used like this Key: SPARK-4067 URL: https://issues.apache.org/jira/browse/SPARK-4067 Project: Spark

[jira] [Commented] (SPARK-4067) refactor ExecutorUncaughtExceptionHandler as a general one as it is used like this

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181980#comment-14181980 ] Apache Spark commented on SPARK-4067: - User 'CodingCat' has created a pull request for

[jira] [Commented] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181988#comment-14181988 ] Apache Spark commented on SPARK-4006: - User 'tsliwowicz' has created a pull request

[jira] [Commented] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182001#comment-14182001 ] Apache Spark commented on SPARK-4006: - User 'tsliwowicz' has created a pull request

[jira] [Commented] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-23 Thread Tal Sliwowicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182005#comment-14182005 ] Tal Sliwowicz commented on SPARK-4006: -- After the fix was merged to master, created a

[jira] [Updated] (SPARK-4068) NPE in jsonRDD schema inference

2014-10-23 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4068: Priority: Critical (was: Major) NPE in jsonRDD schema inference

[jira] [Commented] (SPARK-3278) Isotonic regression

2014-10-23 Thread Martin Zapletal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182064#comment-14182064 ] Martin Zapletal commented on SPARK-3278: I am interested in working on this

[jira] [Created] (SPARK-4069) [SPARK-YARN] ApplicationMaster should releases all executors' containers before unregistering itself from Yarn RM

2014-10-23 Thread Min Zhou (JIRA)
Min Zhou created SPARK-4069: --- Summary: [SPARK-YARN] ApplicationMaster should releases all executors' containers before unregistering itself from Yarn RM Key: SPARK-4069 URL:

[jira] [Updated] (SPARK-4069) [SPARK-YARN] ApplicationMaster should release all executors' containers before unregistering itself from Yarn RM

2014-10-23 Thread Min Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Zhou updated SPARK-4069: Summary: [SPARK-YARN] ApplicationMaster should release all executors' containers before unregistering

[jira] [Created] (SPARK-4070) Clean up web UI's table rendering code

2014-10-23 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4070: - Summary: Clean up web UI's table rendering code Key: SPARK-4070 URL: https://issues.apache.org/jira/browse/SPARK-4070 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-4070) Clean up web UI's table rendering code

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182102#comment-14182102 ] Apache Spark commented on SPARK-4070: - User 'JoshRosen' has created a pull request for

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2014-10-23 Thread Kostas Sakellis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182110#comment-14182110 ] Kostas Sakellis commented on SPARK-1239: Apologies for not commenting on this JIRA

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182122#comment-14182122 ] Patrick Wendell commented on SPARK-1239: Hey Kostas - there are a few other bugs

[jira] [Reopened] (SPARK-3812) Adapt maven build to publish effective pom.

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3812: It appeared that this was creating an issue with the maven tests. I am reverting this to see

[jira] [Commented] (SPARK-2652) Turning default configurations for PySpark

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182159#comment-14182159 ] Apache Spark commented on SPARK-2652: - User 'davies' has created a pull request for

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182175#comment-14182175 ] Patrick Wendell commented on SPARK-4030: I'm fine to open it up. I do think

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-23 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182181#comment-14182181 ] Shivaram Venkataraman commented on SPARK-4030: -- Great -- I'll send a PR and

[jira] [Resolved] (SPARK-4019) Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4019. Resolution: Fixed Fix Version/s: 1.2.0 Fixed by Josh's patch:

[jira] [Created] (SPARK-4071) Unroll fails silently if BlockManager size is small

2014-10-23 Thread Andrew Or (JIRA)
Andrew Or created SPARK-4071: Summary: Unroll fails silently if BlockManager size is small Key: SPARK-4071 URL: https://issues.apache.org/jira/browse/SPARK-4071 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-4071) Unroll fails silently if BlockManager size is small

2014-10-23 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4071: - Description: In tests, we may want to have BlockManagers of size 1MB

[jira] [Commented] (SPARK-4071) Unroll fails silently if BlockManager size is small

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182190#comment-14182190 ] Apache Spark commented on SPARK-4071: - User 'andrewor14' has created a pull request

[jira] [Updated] (SPARK-3278) Isotonic regression

2014-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3278: - Assignee: Martin Zapletal Isotonic regression --- Key:

[jira] [Created] (SPARK-4072) Storage UI does not reflect memory usage by streaming blocks

2014-10-23 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-4072: Summary: Storage UI does not reflect memory usage by streaming blocks Key: SPARK-4072 URL: https://issues.apache.org/jira/browse/SPARK-4072 Project: Spark

[jira] [Commented] (SPARK-4072) Storage UI does not reflect memory usage by streaming blocks

2014-10-23 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182230#comment-14182230 ] Tathagata Das commented on SPARK-4072: -- [~joshrosen] [~andrewor14] How hard would it

[jira] [Resolved] (SPARK-3993) python worker may hang after reused from take()

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3993. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2838

[jira] [Commented] (SPARK-4068) NPE in jsonRDD schema inference

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182251#comment-14182251 ] Apache Spark commented on SPARK-4068: - User 'yhuai' has created a pull request for

[jira] [Updated] (SPARK-4056) Upgrade snappy-java to 1.1.1.5

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4056: -- Description: We should upgrade snappy-java to 1.1.1.5 across all of our maintenance branches. This

[jira] [Updated] (SPARK-4064) If we create too many broadcast variables, the spark has great possibility to hang

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Summary: If we create too many broadcast variables, the spark has great possibility to hang (was: In

[jira] [Updated] (SPARK-4064) If we create too many big broadcast variables, the spark has great possibility to hang

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Summary: If we create too many big broadcast variables, the spark has great possibility to hang

[jira] [Updated] (SPARK-4064) If we create a lot of big broadcast variables, Spark has great possibility to hang

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Summary: If we create a lot of big broadcast variables, Spark has great possibility to hang (was:

[jira] [Updated] (SPARK-4064) If we create a lot of big broadcast variables, Spark has great possibility to hang

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Priority: Critical (was: Major) If we create a lot of big broadcast variables, Spark has great

[jira] [Updated] (SPARK-4064) If we create a lot of big broadcast variables, Spark has great possibility to hang

2014-10-23 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guoqiang Li updated SPARK-4064: --- Affects Version/s: 1.2.0 If we create a lot of big broadcast variables, Spark has great possibility

[jira] [Updated] (SPARK-2663) Support the GroupingSet/ROLLUP/CUBE

2014-10-23 Thread Cheng Hao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Hao updated SPARK-2663: - Attachment: grouping_set.pdf General Design for the implementation of GroupingSet, Cube, Rollup.

[jira] [Commented] (SPARK-3572) Support register UserType in SQL

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182389#comment-14182389 ] Apache Spark commented on SPARK-3572: - User 'jkbradley' has created a pull request for

[jira] [Resolved] (SPARK-4000) Gathers unit tests logs to Jenkins master at the end of a Jenkins build

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4000. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2845

[jira] [Commented] (SPARK-3812) Adapt maven build to publish effective pom.

2014-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182461#comment-14182461 ] Apache Spark commented on SPARK-3812: - User 'ScrapCodes' has created a pull request

[jira] [Created] (SPARK-4073) Parquet+Snappy can cause significant off-heap memory usage

2014-10-23 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4073: -- Summary: Parquet+Snappy can cause significant off-heap memory usage Key: SPARK-4073 URL: https://issues.apache.org/jira/browse/SPARK-4073 Project: Spark