[jira] [Created] (SPARK-19307) SPARK-17387 caused ignorance of conf object passed to SparkContext:

2017-01-19 Thread yuriy_hupalo (JIRA)
yuriy_hupalo created SPARK-19307: Summary: SPARK-17387 caused ignorance of conf object passed to SparkContext: Key: SPARK-19307 URL: https://issues.apache.org/jira/browse/SPARK-19307 Project: Spark

[jira] [Updated] (SPARK-19263) DAGScheduler should avoid sending conflicting task set.

2017-01-19 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19263: - Summary: DAGScheduler should avoid sending conflicting task set. (was: DAGScheduler should handle

[jira] [Created] (SPARK-19306) Fix inconsistent state in DiskBlockObjectWrite when exception occurred

2017-01-19 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-19306: --- Summary: Fix inconsistent state in DiskBlockObjectWrite when exception occurred Key: SPARK-19306 URL: https://issues.apache.org/jira/browse/SPARK-19306 Project: Spark

[jira] [Updated] (SPARK-18116) spark streaming ui show 0 events when recovering from checkpoint

2017-01-19 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-18116: -- Fix Version/s: 2.1.1 2.0.3 > spark streaming ui show 0 events when recovering from

[jira] [Updated] (SPARK-18116) spark streaming ui show 0 events when recovering from checkpoint

2017-01-19 Thread Genmao Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-18116: -- Target Version/s: 2.0.3, 2.1.1 > spark streaming ui show 0 events when recovering from checkpoint >

[jira] [Commented] (SPARK-18116) spark streaming ui show 0 events when recovering from checkpoint

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831345#comment-15831345 ] Apache Spark commented on SPARK-18116: -- User 'uncleGen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18116) spark streaming ui show 0 events when recovering from checkpoint

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18116: Assignee: (was: Apache Spark) > spark streaming ui show 0 events when recovering from

[jira] [Assigned] (SPARK-18116) spark streaming ui show 0 events when recovering from checkpoint

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18116: Assignee: Apache Spark > spark streaming ui show 0 events when recovering from checkpoint

[jira] [Commented] (SPARK-19282) RandomForestRegressionModel should expose getMaxDepth

2017-01-19 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831342#comment-15831342 ] Xin Ren commented on SPARK-19282: - sorry being naive, I'm not familiar with random forest, but is "max

[jira] [Resolved] (SPARK-19271) Change non-cbo estimation of aggregate

2017-01-19 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19271. - Resolution: Fixed Assignee: Zhenhua Wang Fix Version/s: 2.2.0 > Change non-cbo

[jira] [Assigned] (SPARK-19305) partitioned table should always put partition columns at the end of table schema

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19305: Assignee: Wenchen Fan (was: Apache Spark) > partitioned table should always put

[jira] [Commented] (SPARK-19305) partitioned table should always put partition columns at the end of table schema

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831269#comment-15831269 ] Apache Spark commented on SPARK-19305: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19305) partitioned table should always put partition columns at the end of table schema

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19305: Assignee: Apache Spark (was: Wenchen Fan) > partitioned table should always put

[jira] [Created] (SPARK-19305) partitioned table should always put partition columns at the end of table schema

2017-01-19 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19305: --- Summary: partitioned table should always put partition columns at the end of table schema Key: SPARK-19305 URL: https://issues.apache.org/jira/browse/SPARK-19305

[jira] [Updated] (SPARK-19300) Executor is waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Priority: Critical (was: Major) > Executor is waiting for lock > > >

[jira] [Commented] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

2017-01-19 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831217#comment-15831217 ] Nan Zhu commented on SPARK-19280: - BTW, do I need to highlight the KafkaDStream issue as another JIRA,

[jira] [Commented] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

2017-01-19 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831209#comment-15831209 ] Nan Zhu commented on SPARK-19280: - [~zsxwing] Thanks for reply 0) I do not think the content in

[jira] [Created] (SPARK-19304) Kinesis checkpoint recovery is 10x slow

2017-01-19 Thread Gaurav Shah (JIRA)
Gaurav Shah created SPARK-19304: --- Summary: Kinesis checkpoint recovery is 10x slow Key: SPARK-19304 URL: https://issues.apache.org/jira/browse/SPARK-19304 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-19303) Add evaluate method in clustering models

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19303: Assignee: (was: Apache Spark) > Add evaluate method in clustering models >

[jira] [Commented] (SPARK-19303) Add evaluate method in clustering models

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831201#comment-15831201 ] Apache Spark commented on SPARK-19303: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-19303) Add evaluate method in clustering models

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19303: Assignee: Apache Spark > Add evaluate method in clustering models >

[jira] [Created] (SPARK-19303) Add evaluate method in clustering models

2017-01-19 Thread zhengruifeng (JIRA)
zhengruifeng created SPARK-19303: Summary: Add evaluate method in clustering models Key: SPARK-19303 URL: https://issues.apache.org/jira/browse/SPARK-19303 Project: Spark Issue Type:

[jira] [Updated] (SPARK-12347) Write script to run all MLlib examples for testing

2017-01-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-12347: - Shepherd: Felix Cheung > Write script to run all MLlib examples for testing >

[jira] [Commented] (SPARK-12347) Write script to run all MLlib examples for testing

2017-01-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831191#comment-15831191 ] Felix Cheung commented on SPARK-12347: -- Great! > Write script to run all MLlib examples for testing

[jira] [Commented] (SPARK-19302) Fix the wrong item format in security.md

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831188#comment-15831188 ] Apache Spark commented on SPARK-19302: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19302) Fix the wrong item format in security.md

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19302: Assignee: Apache Spark > Fix the wrong item format in security.md >

[jira] [Assigned] (SPARK-19302) Fix the wrong item format in security.md

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19302: Assignee: (was: Apache Spark) > Fix the wrong item format in security.md >

[jira] [Created] (SPARK-19302) Fix the wrong item format in security.md

2017-01-19 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-19302: -- Summary: Fix the wrong item format in security.md Key: SPARK-19302 URL: https://issues.apache.org/jira/browse/SPARK-19302 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831185#comment-15831185 ] Apache Spark commented on SPARK-19234: -- User 'admackin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19234: Assignee: Apache Spark > AFTSurvivalRegression chokes silently or with confusing errors

[jira] [Assigned] (SPARK-19234) AFTSurvivalRegression chokes silently or with confusing errors when any labels are zero

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19234: Assignee: (was: Apache Spark) > AFTSurvivalRegression chokes silently or with

[jira] [Created] (SPARK-19301) SparkContext is ignoring SparkConf when _jvm is not initialized on spark-submit

2017-01-19 Thread Teppei Daito (JIRA)
Teppei Daito created SPARK-19301: Summary: SparkContext is ignoring SparkConf when _jvm is not initialized on spark-submit Key: SPARK-19301 URL: https://issues.apache.org/jira/browse/SPARK-19301

[jira] [Resolved] (SPARK-19292) filter with partition columns should be case-insensitive on Hive tables

2017-01-19 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19292. - Resolution: Fixed > filter with partition columns should be case-insensitive on Hive tables >

[jira] [Updated] (SPARK-19292) filter with partition columns should be case-insensitive on Hive tables

2017-01-19 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19292: Fix Version/s: 2.2.0 > filter with partition columns should be case-insensitive on Hive tables >

[jira] [Updated] (SPARK-19300) Executor is waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Description: I can see all threads in the executor is waiting for lock for a long time. And then it

[jira] [Updated] (SPARK-19300) Executor is waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Description: I can see all threads in the executor is waiting for lock for a long time. And then it

[jira] [Updated] (SPARK-19300) Executor hang waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Description: I can see the executor are wa {code} sun.misc.Unsafe.park(Native Method)

[jira] [Updated] (SPARK-19300) Executor hang waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Priority: Major (was: Critical) > Executor hang waiting for lock > -- > >

[jira] [Updated] (SPARK-19300) Executor is waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Summary: Executor is waiting for lock (was: Executor hang waiting for lock) > Executor is waiting for

[jira] [Updated] (SPARK-19300) Executor hang waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Description: I can see {code} sun.misc.Unsafe.park(Native Method)

[jira] [Updated] (SPARK-19300) Executor hang waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai updated SPARK-19300: -- Component/s: Spark Core > Executor hang waiting for lock > -- > >

[jira] [Created] (SPARK-19300) Executor hang waiting for lock

2017-01-19 Thread cen yuhai (JIRA)
cen yuhai created SPARK-19300: - Summary: Executor hang waiting for lock Key: SPARK-19300 URL: https://issues.apache.org/jira/browse/SPARK-19300 Project: Spark Issue Type: Bug Affects

[jira] [Updated] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

2017-01-19 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nan Zhu updated SPARK-19280: Description: In one of our applications, we found the following issue, the application recovering from a

[jira] [Commented] (SPARK-19233) Inconsistent Behaviour of Spark Streaming Checkpoint

2017-01-19 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831098#comment-15831098 ] Nan Zhu commented on SPARK-19233: - By filtering generatedRDDs, I may bring some confusion here, what I

[jira] [Updated] (SPARK-19299) Nulls in non nullable columns causes data corruption in parquet

2017-01-19 Thread Franklyn Dsouza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklyn Dsouza updated SPARK-19299: Description: The problem we're seeing is that if a null occurs in a no-nullable field and

[jira] [Updated] (SPARK-19299) Nulls in non nullable columns causes data corruption in parquet

2017-01-19 Thread Franklyn Dsouza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklyn Dsouza updated SPARK-19299: Description: The problem we're seeing is that if a null occurs in a no-nullable field and

[jira] [Updated] (SPARK-19299) Nulls in non nullable columns causes data corruption in parquet

2017-01-19 Thread Franklyn Dsouza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklyn Dsouza updated SPARK-19299: Description: The problem we're seeing is that if a null occurs in a no-nullable field and

[jira] [Updated] (SPARK-19299) Nulls in non nullable columns causes data corruption in parquet

2017-01-19 Thread Franklyn Dsouza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklyn Dsouza updated SPARK-19299: Description: The problem we're seeing is that if a null occurs in a no-nullable field and

[jira] [Updated] (SPARK-19299) Nulls in non nullable columns causes data corruption in parquet

2017-01-19 Thread Franklyn Dsouza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklyn Dsouza updated SPARK-19299: Description: The problem we're seeing is that if a null occurs in a no-nullable field and

[jira] [Created] (SPARK-19299) Nulls in non nullable columns causes data corruption in parquet

2017-01-19 Thread Franklyn Dsouza (JIRA)
Franklyn Dsouza created SPARK-19299: --- Summary: Nulls in non nullable columns causes data corruption in parquet Key: SPARK-19299 URL: https://issues.apache.org/jira/browse/SPARK-19299 Project: Spark

[jira] [Assigned] (SPARK-19298) History server can't match MalformedInputException and prompt the detail logs while repalying eventlog

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19298: Assignee: (was: Apache Spark) > History server can't match MalformedInputException

[jira] [Commented] (SPARK-19298) History server can't match MalformedInputException and prompt the detail logs while repalying eventlog

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831046#comment-15831046 ] Apache Spark commented on SPARK-19298: -- User 'sharkdtu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19298) History server can't match MalformedInputException and prompt the detail logs while repalying eventlog

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19298: Assignee: Apache Spark > History server can't match MalformedInputException and prompt

[jira] [Created] (SPARK-19298) History server can't match MalformedInputException and prompt the detail logs while repalying eventlog

2017-01-19 Thread sharkd tu (JIRA)
sharkd tu created SPARK-19298: - Summary: History server can't match MalformedInputException and prompt the detail logs while repalying eventlog Key: SPARK-19298 URL: https://issues.apache.org/jira/browse/SPARK-19298

[jira] [Updated] (SPARK-19298) History server can't match MalformedInputException and prompt the detail logs while repalying eventlog

2017-01-19 Thread sharkd tu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated SPARK-19298: -- Description: Could't match MalformedInputException and prompt the detail logs while repalying

[jira] [Commented] (SPARK-19296) Awkward changes for JdbcUtils.saveTable in Spark 2.1.0

2017-01-19 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831033#comment-15831033 ] Paul Wu commented on SPARK-19296: - We found this Util is very useful in general (much, much better than

[jira] [Commented] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

2017-01-19 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831018#comment-15831018 ] Shixiong Zhu commented on SPARK-19280: -- Good catch and nice explanation. I think maybe 2) is the

[jira] [Updated] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

2017-01-19 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19280: - Priority: Critical (was: Major) > Failed Recovery from checkpoint caused by the multi-threads

[jira] [Assigned] (SPARK-16554) Spark should kill executors when they are blacklisted

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16554: Assignee: Jose Soltren (was: Apache Spark) > Spark should kill executors when they are

[jira] [Assigned] (SPARK-16554) Spark should kill executors when they are blacklisted

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16554: Assignee: Apache Spark (was: Jose Soltren) > Spark should kill executors when they are

[jira] [Commented] (SPARK-16554) Spark should kill executors when they are blacklisted

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831005#comment-15831005 ] Apache Spark commented on SPARK-16554: -- User 'jsoltren' has created a pull request for this issue:

[jira] [Commented] (SPARK-19283) Application details UI not visible for completed actions

2017-01-19 Thread Brian Cantoni (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830999#comment-15830999 ] Brian Cantoni commented on SPARK-19283: --- For reference, the story which made that change in 2.0 was

[jira] [Commented] (SPARK-19233) Inconsistent Behaviour of Spark Streaming Checkpoint

2017-01-19 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830989#comment-15830989 ] Shixiong Zhu commented on SPARK-19233: -- I don't think filtering "generatedRDDs" will work.

[jira] [Commented] (SPARK-19275) Spark Streaming, Kafka receiver, "Failed to get records for ... after polling for 512"

2017-01-19 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830976#comment-15830976 ] Shixiong Zhu commented on SPARK-19275: -- This error usually means Spark cannot fetch records from

[jira] [Commented] (SPARK-19282) RandomForestRegressionModel should expose getMaxDepth

2017-01-19 Thread Nick Lothian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830962#comment-15830962 ] Nick Lothian commented on SPARK-19282: -- The docs say it is available in Java and Scala. In Java:

[jira] [Commented] (SPARK-19296) Awkward changes for JdbcUtils.saveTable in Spark 2.1.0

2017-01-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830924#comment-15830924 ] Hyukjin Kwon commented on SPARK-19296: -- {quote} incompatible to previous versions {quote} If this

[jira] [Comment Edited] (SPARK-19296) Awkward changes for JdbcUtils.saveTable in Spark 2.1.0

2017-01-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830915#comment-15830915 ] Hyukjin Kwon edited comment on SPARK-19296 at 1/20/17 12:53 AM:

[jira] [Commented] (SPARK-19296) Awkward changes for JdbcUtils.saveTable in Spark 2.1.0

2017-01-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830915#comment-15830915 ] Hyukjin Kwon commented on SPARK-19296: --

[jira] [Commented] (SPARK-17436) dataframe.write sometimes does not keep sorting

2017-01-19 Thread Jason Moore (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830889#comment-15830889 ] Jason Moore commented on SPARK-17436: - Hi [~ran.h...@optimalplus.com], [~srowen], How sure are we

[jira] [Resolved] (SPARK-17912) Refactor code generation to get data for ColumnVector/ColumnarBatch

2017-01-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17912. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 15467

[jira] [Updated] (SPARK-17912) Refactor code generation to get data for ColumnVector/ColumnarBatch

2017-01-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-17912: --- Fix Version/s: (was: 3.0.0) 2.2.0 > Refactor code generation to get data for

[jira] [Commented] (SPARK-14141) Let user specify datatypes of pandas dataframe in toPandas()

2017-01-19 Thread Luke Miner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830750#comment-15830750 ] Luke Miner commented on SPARK-14141: One option is to convert all the categorical variables into

[jira] [Comment Edited] (SPARK-14141) Let user specify datatypes of pandas dataframe in toPandas()

2017-01-19 Thread Luke Miner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830750#comment-15830750 ] Luke Miner edited comment on SPARK-14141 at 1/19/17 10:52 PM: -- One option is

[jira] [Commented] (SPARK-19287) JavaPairRDD flatMapValues requires function returning Iterable, not Iterator

2017-01-19 Thread Asher Krim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830715#comment-15830715 ] Asher Krim commented on SPARK-19287: Considering that this was an oversight and should have been

[jira] [Resolved] (SPARK-19295) IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars

2017-01-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-19295. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16649

[jira] [Commented] (SPARK-19268) File does not exist: /tmp/temporary-157b89c1-27bb-49f3-a70c-ca1b75022b4d/state/0/2/1.delta

2017-01-19 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830706#comment-15830706 ] Shixiong Zhu commented on SPARK-19268: -- Right now Structured Streaming doesn't support

[jira] [Updated] (SPARK-19268) File does not exist: /tmp/temporary-157b89c1-27bb-49f3-a70c-ca1b75022b4d/state/0/2/1.delta

2017-01-19 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19268: - Priority: Critical (was: Major) > File does not exist: >

[jira] [Created] (SPARK-19297) Add ability for --packages tag to pull latest version

2017-01-19 Thread Steven Landes (JIRA)
Steven Landes created SPARK-19297: - Summary: Add ability for --packages tag to pull latest version Key: SPARK-19297 URL: https://issues.apache.org/jira/browse/SPARK-19297 Project: Spark

[jira] [Commented] (SPARK-17602) PySpark - Performance Optimization Large Size of Broadcast Variable

2017-01-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830644#comment-15830644 ] Davies Liu commented on SPARK-17602: The Python workers are reused by default, could you re-run the

[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods

2017-01-19 Thread Salil Surendran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830630#comment-15830630 ] Salil Surendran commented on SPARK-18120: - [~thomastechs]I will be making a PR today. >

[jira] [Commented] (SPARK-19295) IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830560#comment-15830560 ] Apache Spark commented on SPARK-19295: -- User 'yhuai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19295) IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19295: Assignee: Yin Huai (was: Apache Spark) > IsolatedClientLoader's downloadVersion should

[jira] [Assigned] (SPARK-19295) IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars

2017-01-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19295: Assignee: Apache Spark (was: Yin Huai) > IsolatedClientLoader's downloadVersion should

[jira] [Created] (SPARK-19296) Awkward changes for JdbcUtils.saveTable in Spark 2.1.0

2017-01-19 Thread Paul Wu (JIRA)
Paul Wu created SPARK-19296: --- Summary: Awkward changes for JdbcUtils.saveTable in Spark 2.1.0 Key: SPARK-19296 URL: https://issues.apache.org/jira/browse/SPARK-19296 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-19295) IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars

2017-01-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-19295: - Priority: Minor (was: Major) > IsolatedClientLoader's downloadVersion should log the location of

[jira] [Updated] (SPARK-19295) IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars

2017-01-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-19295: - Issue Type: Improvement (was: Bug) > IsolatedClientLoader's downloadVersion should log the location of

[jira] [Created] (SPARK-19295) IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars

2017-01-19 Thread Yin Huai (JIRA)
Yin Huai created SPARK-19295: Summary: IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars Key: SPARK-19295 URL: https://issues.apache.org/jira/browse/SPARK-19295

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2017-01-19 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830530#comment-15830530 ] Imran Rashid commented on SPARK-18886: -- I had another idea for how to fix this. In addition to

[jira] [Commented] (SPARK-17602) PySpark - Performance Optimization Large Size of Broadcast Variable

2017-01-19 Thread Junfeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830528#comment-15830528 ] Junfeng commented on SPARK-17602: - Thanks [~holdenk] [~davies] Could you let me know your comments

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

2017-01-19 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830519#comment-15830519 ] Mark Hamstra commented on SPARK-19276: -- Ok, I haven't read your PR closely yet, so I missed that.

[jira] [Reopened] (SPARK-13478) Fetching delegation tokens for Hive fails when using proxy users

2017-01-19 Thread John Muller (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Muller reopened SPARK-13478: - Can we get this patch applied to 1.6.x as well? > Fetching delegation tokens for Hive fails when

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

2017-01-19 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830487#comment-15830487 ] Imran Rashid commented on SPARK-19276: -- [~markhamstra] bq. I guess my only real question is if we

[jira] [Created] (SPARK-19294) improve ml LDA save/load

2017-01-19 Thread Asher Krim (JIRA)
Asher Krim created SPARK-19294: -- Summary: improve ml LDA save/load Key: SPARK-19294 URL: https://issues.apache.org/jira/browse/SPARK-19294 Project: Spark Issue Type: Bug Reporter:

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

2017-01-19 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830387#comment-15830387 ] Mark Hamstra commented on SPARK-19276: -- This all makes sense, and the PR is a good effort to fix

[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods

2017-01-19 Thread Thomas Sebastian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830307#comment-15830307 ] Thomas Sebastian commented on SPARK-18120: -- Hi [~salilsurendran] Are you working on this, please

[jira] [Commented] (SPARK-18406) Race between end-of-task and completion iterator read lock release

2017-01-19 Thread John Myers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830294#comment-15830294 ] John Myers commented on SPARK-18406: Similar issue in doing basic RDD operations (like checking for

[jira] [Commented] (SPARK-19288) Failure (at test_sparkSQL.R#1300): date functions on a DataFrame in R/run-tests.sh

2017-01-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830284#comment-15830284 ] Felix Cheung commented on SPARK-19288: -- We are not seeing this in Jenkins? Which branch are you

[jira] [Commented] (SPARK-18496) java.lang.AssertionError: assertion failed

2017-01-19 Thread John Myers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830278#comment-15830278 ] John Myers commented on SPARK-18496: I get the same error while trying to do RDD operations off of a

[jira] [Commented] (SPARK-17557) SQL query on parquet table java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary

2017-01-19 Thread Jayadevan M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830247#comment-15830247 ] Jayadevan M commented on SPARK-17557: - Hi [~epahomov] I tried to replicate this issue using below

[jira] [Resolved] (SPARK-19283) Application details UI not visible for completed actions

2017-01-19 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-19283. -- Resolution: Not A Problem > Application details UI not visible for completed actions >

  1   2   >