[jira] [Updated] (SPARK-20135) spark thriftserver2: no job running but cores not release on yarn

2017-03-28 Thread bruce xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bruce xu updated SPARK-20135: - Attachment: 0329-3.png 0329-2.png 0329-1.png cores and memory not

[jira] [Created] (SPARK-20135) spark thriftserver2: no job running but cores not release on yarn

2017-03-28 Thread bruce xu (JIRA)
bruce xu created SPARK-20135: Summary: spark thriftserver2: no job running but cores not release on yarn Key: SPARK-20135 URL: https://issues.apache.org/jira/browse/SPARK-20135 Project: Spark

[jira] [Updated] (SPARK-20107) Add spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version option to configuration.md

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Description: Set {{spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2}} can speed up

[jira] [Assigned] (SPARK-20107) Add spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version option to configuration.md

2017-03-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20107: - Assignee: Yuming Wang Priority: Trivial (was: Major) Summary: Add

[jira] [Updated] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Component/s: (was: SQL) > Speed up HadoopMapReduceCommitProtocol#commitJob for many output

[jira] [Updated] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Component/s: Documentation > Speed up HadoopMapReduceCommitProtocol#commitJob for many output

[jira] [Assigned] (SPARK-20134) SQLMetrics.postDriverMetricUpdates to simplify driver side metric updates

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20134: Assignee: Reynold Xin (was: Apache Spark) > SQLMetrics.postDriverMetricUpdates to

[jira] [Assigned] (SPARK-20134) SQLMetrics.postDriverMetricUpdates to simplify driver side metric updates

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20134: Assignee: Apache Spark (was: Reynold Xin) > SQLMetrics.postDriverMetricUpdates to

[jira] [Commented] (SPARK-20134) SQLMetrics.postDriverMetricUpdates to simplify driver side metric updates

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946525#comment-15946525 ] Apache Spark commented on SPARK-20134: -- User 'rxin' has created a pull request for this issue:

[jira] [Created] (SPARK-20134) SQLMetrics.postDriverMetricUpdates to simplify driver side metric updates

2017-03-28 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-20134: --- Summary: SQLMetrics.postDriverMetricUpdates to simplify driver side metric updates Key: SPARK-20134 URL: https://issues.apache.org/jira/browse/SPARK-20134 Project:

[jira] [Assigned] (SPARK-20131) Flaky Test: org.apache.spark.streaming.StreamingContextSuite

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20131: Assignee: (was: Apache Spark) > Flaky Test:

[jira] [Commented] (SPARK-20131) Flaky Test: org.apache.spark.streaming.StreamingContextSuite

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946522#comment-15946522 ] Apache Spark commented on SPARK-20131: -- User 'uncleGen' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20093) Exception when Joining dataframe with another dataframe generated by applying groupBy transformation on original one

2017-03-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20093. -- Resolution: Duplicate ^ It seems a duplicate of that to me as well. I am resolving this.

[jira] [Commented] (SPARK-20093) Exception when Joining dataframe with another dataframe generated by applying groupBy transformation on original one

2017-03-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946501#comment-15946501 ] Takeshi Yamamuro commented on SPARK-20093: -- It seems this issue is the same with SPARK-10925. >

[jira] [Commented] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946493#comment-15946493 ] Saisai Shao commented on SPARK-20128: - Sorry I cannot access the logs. What I could see from the link

[jira] [Commented] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946486#comment-15946486 ] Imran Rashid commented on SPARK-20128: -- Thanks [~jerryshao], that is helpful, definitely good to

[jira] [Comment Edited] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946457#comment-15946457 ] Saisai Shao edited comment on SPARK-20128 at 3/29/17 3:20 AM: -- Here the

[jira] [Commented] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946457#comment-15946457 ] Saisai Shao commented on SPARK-20128: - Here the exception is from MasterSource, which only exists in

[jira] [Updated] (SPARK-20133) User guide for spark.ml.stat.ChiSquareTest

2017-03-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-20133: -- Description: Add new user guide section for spark.ml.stat, and document ChiSquareTest.

[jira] [Created] (SPARK-20133) User guide for spark.ml.stat.ChiSquareTest

2017-03-28 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-20133: - Summary: User guide for spark.ml.stat.ChiSquareTest Key: SPARK-20133 URL: https://issues.apache.org/jira/browse/SPARK-20133 Project: Spark Issue

[jira] [Resolved] (SPARK-20040) Python API for ml.stat.ChiSquareTest

2017-03-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-20040. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17421

[jira] [Assigned] (SPARK-20040) Python API for ml.stat.ChiSquareTest

2017-03-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-20040: - Assignee: Joseph K. Bradley > Python API for ml.stat.ChiSquareTest >

[jira] [Assigned] (SPARK-20040) Python API for ml.stat.ChiSquareTest

2017-03-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-20040: - Assignee: Bago Amirbekian (was: Joseph K. Bradley) > Python API for

[jira] [Comment Edited] (SPARK-20093) Exception when Joining dataframe with another dataframe generated by applying groupBy transformation on original one

2017-03-28 Thread Yong Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946375#comment-15946375 ] Yong Zhang edited comment on SPARK-20093 at 3/29/17 1:34 AM: - This problem

[jira] [Commented] (SPARK-20093) Exception when Joining dataframe with another dataframe generated by applying groupBy transformation on original one

2017-03-28 Thread Yong Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946375#comment-15946375 ] Yong Zhang commented on SPARK-20093: This problem exists. It looks like if switch the order of join,

[jira] [Commented] (SPARK-16288) Implement inline table generating function

2017-03-28 Thread Guilherme Braccialli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946176#comment-15946176 ] Guilherme Braccialli commented on SPARK-16288: -- Is it possible to call this function direct

[jira] [Resolved] (SPARK-20043) Decision Tree loader does not handle uppercase impurity param values

2017-03-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-20043. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Issue

[jira] [Assigned] (SPARK-20043) Decision Tree loader does not handle uppercase impurity param values

2017-03-28 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-20043: - Assignee: Yan Facai (颜发才) > Decision Tree loader does not handle uppercase

[jira] [Commented] (SPARK-20132) Add documentation for column string functions

2017-03-28 Thread Michael Patterson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946130#comment-15946130 ] Michael Patterson commented on SPARK-20132: --- I have a commit with the documentation:

[jira] [Created] (SPARK-20132) Add documentation for column string functions

2017-03-28 Thread Michael Patterson (JIRA)
Michael Patterson created SPARK-20132: - Summary: Add documentation for column string functions Key: SPARK-20132 URL: https://issues.apache.org/jira/browse/SPARK-20132 Project: Spark

[jira] [Assigned] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20050: Assignee: (was: Apache Spark) > Kafka 0.10 DirectStream doesn't commit last processed

[jira] [Assigned] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20050: Assignee: Apache Spark > Kafka 0.10 DirectStream doesn't commit last processed batch's

[jira] [Commented] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946098#comment-15946098 ] Apache Spark commented on SPARK-20050: -- User 'sasakitoa' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20125) Dataset of type option of map does not work

2017-03-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-20125. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Dataset of type option of map

[jira] [Updated] (SPARK-14536) NPE in JDBCRDD when array column contains nulls (postgresql)

2017-03-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-14536: Fix Version/s: 2.1.1 > NPE in JDBCRDD when array column contains nulls (postgresql) >

[jira] [Comment Edited] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-03-28 Thread Mathieu D (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D edited comment on SPARK-20082 at 3/28/17 8:39 PM: [~yuhaoyan] would

[jira] [Commented] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-03-28 Thread Mathieu D (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D commented on SPARK-20082: --- [~yuhaoyan] would you mind having a look to this PR. Right now, I

[jira] [Comment Edited] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-03-28 Thread Mathieu D (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D edited comment on SPARK-20082 at 3/28/17 8:26 PM: [~yuhaoyan] would

[jira] [Comment Edited] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-03-28 Thread Mathieu D (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D edited comment on SPARK-20082 at 3/28/17 8:27 PM: [~yuhaoyan] would

[jira] [Assigned] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20082: Assignee: (was: Apache Spark) > Incremental update of LDA model, by adding

[jira] [Assigned] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20082: Assignee: Apache Spark > Incremental update of LDA model, by adding initialModel as start

[jira] [Commented] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945866#comment-15945866 ] Apache Spark commented on SPARK-20082: -- User 'mdespriee' has created a pull request for this issue:

[jira] [Updated] (SPARK-16929) Speculation-related synchronization bottleneck in checkSpeculatableTasks

2017-03-28 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-16929: - Issue Type: Improvement (was: Bug) > Speculation-related synchronization bottleneck in

[jira] [Assigned] (SPARK-19868) conflict TasksetManager lead to spark stopped

2017-03-28 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout reassigned SPARK-19868: -- Assignee: liujianhui > conflict TasksetManager lead to spark stopped >

[jira] [Resolved] (SPARK-19868) conflict TasksetManager lead to spark stopped

2017-03-28 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout resolved SPARK-19868. Resolution: Fixed Fix Version/s: 2.2.0 > conflict TasksetManager lead to spark

[jira] [Created] (SPARK-20131) Flaky Test: org.apache.spark.streaming.StreamingContextSuite

2017-03-28 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-20131: - Summary: Flaky Test: org.apache.spark.streaming.StreamingContextSuite Key: SPARK-20131 URL: https://issues.apache.org/jira/browse/SPARK-20131 Project: Spark

[jira] [Commented] (SPARK-19551) Theme for PySpark documenation could do with improving

2017-03-28 Thread Arthur Tacca (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945685#comment-15945685 ] Arthur Tacca commented on SPARK-19551: -- Thanks, I needed the reminder! In fact the person that

[jira] [Commented] (SPARK-14536) NPE in JDBCRDD when array column contains nulls (postgresql)

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945673#comment-15945673 ] Apache Spark commented on SPARK-14536: -- User 'sureshthalamati' has created a pull request for this

[jira] [Created] (SPARK-20130) Flaky test: BlockManagerProactiveReplicationSuite

2017-03-28 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-20130: -- Summary: Flaky test: BlockManagerProactiveReplicationSuite Key: SPARK-20130 URL: https://issues.apache.org/jira/browse/SPARK-20130 Project: Spark Issue

[jira] [Resolved] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2017-03-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19995. Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-28 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1594#comment-1594 ] Kazuaki Ishizaki commented on SPARK-20112: -- [~MasterDDT] Thank you for preparing additional

[jira] [Created] (SPARK-20129) JavaSparkContext should use SparkContext.getOrCreate

2017-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-20129: - Summary: JavaSparkContext should use SparkContext.getOrCreate Key: SPARK-20129 URL: https://issues.apache.org/jira/browse/SPARK-20129 Project: Spark Issue

[jira] [Assigned] (SPARK-20129) JavaSparkContext should use SparkContext.getOrCreate

2017-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-20129: - Assignee: Xiangrui Meng > JavaSparkContext should use SparkContext.getOrCreate >

[jira] [Comment Edited] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945399#comment-15945399 ] Mitesh edited comment on SPARK-20112 at 3/28/17 3:46 PM: - [~kiszk] I can try out

[jira] [Comment Edited] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945399#comment-15945399 ] Mitesh edited comment on SPARK-20112 at 3/28/17 3:46 PM: - [~kiszk] I can try out

[jira] [Comment Edited] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945399#comment-15945399 ] Mitesh edited comment on SPARK-20112 at 3/28/17 3:40 PM: - [~kiszk] I can try out

[jira] [Commented] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945399#comment-15945399 ] Mitesh commented on SPARK-20112: [~kiszk] I can try out spark 2.0.3+ or 2.1. Actually I disabled

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Attachment: hs_err_pid22870.log > SIGSEGV in GeneratedIterator.sort_addToSorter >

[jira] [Comment Edited] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945379#comment-15945379 ] Yuming Wang edited comment on SPARK-20107 at 3/28/17 3:37 PM: -- OK, I will

[jira] [Assigned] (SPARK-20109) Need a way to convert from IndexedRowMatrix to Dense Block Matrices

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20109: Assignee: (was: Apache Spark) > Need a way to convert from IndexedRowMatrix to Dense

[jira] [Updated] (SPARK-20109) Need a way to convert from IndexedRowMatrix to Dense Block Matrices

2017-03-28 Thread John Compitello (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Compitello updated SPARK-20109: Description: The current implementation of toBlockMatrix on IndexedRowMatrix is

[jira] [Commented] (SPARK-20109) Need a way to convert from IndexedRowMatrix to Dense Block Matrices

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945388#comment-15945388 ] Apache Spark commented on SPARK-20109: -- User 'johnc1231' has created a pull request for this issue:

[jira] [Commented] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-28 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945389#comment-15945389 ] Kazuaki Ishizaki commented on SPARK-20112: -- SPARK-18745 fixed integer overflow issues in

[jira] [Assigned] (SPARK-20109) Need a way to convert from IndexedRowMatrix to Dense Block Matrices

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20109: Assignee: Apache Spark > Need a way to convert from IndexedRowMatrix to Dense Block

[jira] [Updated] (SPARK-20109) Need a way to convert from IndexedRowMatrix to Dense Block Matrices

2017-03-28 Thread John Compitello (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Compitello updated SPARK-20109: Summary: Need a way to convert from IndexedRowMatrix to Dense Block Matrices (was: Need a

[jira] [Commented] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945379#comment-15945379 ] Yuming Wang commented on SPARK-20107: - OK, I will add

[jira] [Comment Edited] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945166#comment-15945166 ] Mitesh edited comment on SPARK-19981 at 3/28/17 3:21 PM: - As I mentioned on the

[jira] [Resolved] (SPARK-20126) Remove HiveSessionState

2017-03-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20126. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17457

[jira] [Resolved] (SPARK-20124) Join reorder should keep the same order of final project attributes

2017-03-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20124. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17453

[jira] [Assigned] (SPARK-20124) Join reorder should keep the same order of final project attributes

2017-03-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20124: --- Assignee: Zhenhua Wang > Join reorder should keep the same order of final project

[jira] [Commented] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945177#comment-15945177 ] Imran Rashid commented on SPARK-20107: -- >From Marcelo's comment on the PR: bq. We shouldn't set the

[jira] [Comment Edited] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945166#comment-15945166 ] Mitesh edited comment on SPARK-19981 at 3/28/17 1:35 PM: - As I mentioned on the

[jira] [Comment Edited] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945166#comment-15945166 ] Mitesh edited comment on SPARK-19981 at 3/28/17 1:34 PM: - As I mentioned on the

[jira] [Comment Edited] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945166#comment-15945166 ] Mitesh edited comment on SPARK-19981 at 3/28/17 1:34 PM: - As I mentioned on the

[jira] [Comment Edited] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945166#comment-15945166 ] Mitesh edited comment on SPARK-19981 at 3/28/17 1:34 PM: - As I mentioned on the

[jira] [Comment Edited] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945166#comment-15945166 ] Mitesh edited comment on SPARK-19981 at 3/28/17 1:33 PM: - As I mentioned on the

[jira] [Commented] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

2017-03-28 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945166#comment-15945166 ] Mitesh commented on SPARK-19981: As I mentioned on the PR, this seems like it should be handled here:

[jira] [Updated] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-20128: - Description: One Jenkins run failed due to the MetricsSystem never getting killed after a

[jira] [Created] (SPARK-20128) MetricsSystem not always killed in SparkContext.stop()

2017-03-28 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-20128: Summary: MetricsSystem not always killed in SparkContext.stop() Key: SPARK-20128 URL: https://issues.apache.org/jira/browse/SPARK-20128 Project: Spark Issue

[jira] [Assigned] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20127: Assignee: Apache Spark > Minor code cleanup > -- > > Key:

[jira] [Assigned] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20127: Assignee: (was: Apache Spark) > Minor code cleanup > -- > >

[jira] [Commented] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945107#comment-15945107 ] Apache Spark commented on SPARK-20127: -- User 'dbolshak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20126) Remove HiveSessionState

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20126: Assignee: Herman van Hovell (was: Apache Spark) > Remove HiveSessionState >

[jira] [Assigned] (SPARK-20126) Remove HiveSessionState

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20126: Assignee: Apache Spark (was: Herman van Hovell) > Remove HiveSessionState >

[jira] [Commented] (SPARK-20126) Remove HiveSessionState

2017-03-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945072#comment-15945072 ] Apache Spark commented on SPARK-20126: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Commented] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945058#comment-15945058 ] Sean Owen commented on SPARK-20127: --- We use pull requests to suggest changes, but before you do, I

[jira] [Comment Edited] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Denis Bolshakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945051#comment-15945051 ] Denis Bolshakov edited comment on SPARK-20127 at 3/28/17 12:32 PM: --- I

[jira] [Commented] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Denis Bolshakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945051#comment-15945051 ] Denis Bolshakov commented on SPARK-20127: - I applied you first comments (just by reverting

[jira] [Commented] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Denis Bolshakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945014#comment-15945014 ] Denis Bolshakov commented on SPARK-20127: - You can review changes shortly here

[jira] [Updated] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20127: -- Affects Version/s: (was: 2.3.0) 2.1.0 Labels: (was:

[jira] [Comment Edited] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Denis Bolshakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945002#comment-15945002 ] Denis Bolshakov edited comment on SPARK-20127 at 3/28/17 11:46 AM: ---

[jira] [Commented] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Denis Bolshakov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945002#comment-15945002 ] Denis Bolshakov commented on SPARK-20127: - Hello [~srowen], thanks for quick feedback. Could you

[jira] [Resolved] (SPARK-20094) Should Prevent push down of IN subquery to Join operator

2017-03-28 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-20094. --- Resolution: Fixed Assignee: Zhenhua Wang Fix Version/s: 2.2.0 >

[jira] [Commented] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944984#comment-15944984 ] Sean Owen commented on SPARK-20127: --- I am a fan of improving code style and static inspection. However

[jira] [Created] (SPARK-20127) Minor code cleanup

2017-03-28 Thread Denis Bolshakov (JIRA)
Denis Bolshakov created SPARK-20127: --- Summary: Minor code cleanup Key: SPARK-20127 URL: https://issues.apache.org/jira/browse/SPARK-20127 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-20126) Remove HiveSessionState

2017-03-28 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-20126: - Summary: Remove HiveSessionState Key: SPARK-20126 URL: https://issues.apache.org/jira/browse/SPARK-20126 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-20123) $SPARK_HOME variable might have spaces in it(e.g. $SPARK_HOME=/home/spark build/spark), then build spark failed.

2017-03-28 Thread zuotingbing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-20123: Description: If $SPARK_HOME or $FWDIR variable contains spaces, then use

[jira] [Commented] (SPARK-14228) Lost executor of RPC disassociated, and occurs exception: Could not find CoarseGrainedScheduler or it has been stopped

2017-03-28 Thread Amitabh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944912#comment-15944912 ] Amitabh commented on SPARK-14228: - Hi, can you specify the version you were working with? I have received

[jira] [Commented] (SPARK-10294) When Parquet writer's close method throws an exception, we will call close again and trigger a NPE

2017-03-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944887#comment-15944887 ] Steve Loughran commented on SPARK-10294: consider it a failure in the exception logic; it tries

[jira] [Updated] (SPARK-20094) Should Prevent push down of IN subquery to Join operator

2017-03-28 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20094: - Summary: Should Prevent push down of IN subquery to Join operator (was: Putting predicate with

  1   2   >