[jira] [Assigned] (SPARK-19571) appveyor windows tests are failing

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19571: Assignee: (was: Apache Spark) > appveyor windows tests are failing >

[jira] [Commented] (SPARK-19571) appveyor windows tests are failing

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865898#comment-15865898 ] Apache Spark commented on SPARK-19571: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-19571) appveyor windows tests are failing

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19571: Assignee: Apache Spark > appveyor windows tests are failing >

[jira] [Closed] (SPARK-19484) continue work to create a table with an empty schema

2017-02-14 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Song Jun closed SPARK-19484. Resolution: Won't Fix this has been contained in https://github.com/apache/spark/pull/16787 > continue

[jira] [Closed] (SPARK-19491) add a config for tableRelation cache size in SessionCatalog

2017-02-14 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Song Jun closed SPARK-19491. Resolution: Duplicate duplicate with https://github.com/apache/spark/pull/16736 > add a config for

[jira] [Resolved] (SPARK-19453) Correct DataFrame.replace docs

2017-02-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-19453. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16792

[jira] [Assigned] (SPARK-18541) Add pyspark.sql.Column.aliasWithMetadata to allow dynamic metadata management in pyspark SQL API

2017-02-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-18541: --- Assignee: Shea Parkes > Add pyspark.sql.Column.aliasWithMetadata to allow dynamic metadata

[jira] [Resolved] (SPARK-18541) Add pyspark.sql.Column.aliasWithMetadata to allow dynamic metadata management in pyspark SQL API

2017-02-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-18541. - Resolution: Fixed Fix Version/s: 2.2.0 > Add pyspark.sql.Column.aliasWithMetadata to allow

[jira] [Updated] (SPARK-19571) tests are failing to run on Windows with another instance Derby error with Hadoop 2.6.5

2017-02-14 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-19571: - Summary: tests are failing to run on Windows with another instance Derby error with Hadoop 2.6.5

[jira] [Commented] (SPARK-12957) Derive and propagate data constrains in logical plan

2017-02-14 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866239#comment-15866239 ] Nick Dimiduk commented on SPARK-12957: -- [~sameerag] thanks for the comment. From a naive scan of the

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2017-02-14 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866262#comment-15866262 ] Nick Dimiduk commented on SPARK-13219: -- I would implement this manually by materializing the smaller

[jira] [Commented] (SPARK-4591) Algorithm/model parity for spark.ml (Scala)

2017-02-14 Thread Timothy Hunter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866288#comment-15866288 ] Timothy Hunter commented on SPARK-4591: --- [~josephkb] do you also want some subtasks for

[jira] [Assigned] (SPARK-19599) Clean up HDFSMetadataLog for Hadoop 2.6+

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19599: Assignee: (was: Apache Spark) > Clean up HDFSMetadataLog for Hadoop 2.6+ >

[jira] [Assigned] (SPARK-19599) Clean up HDFSMetadataLog for Hadoop 2.6+

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19599: Assignee: Apache Spark > Clean up HDFSMetadataLog for Hadoop 2.6+ >

[jira] [Commented] (SPARK-19599) Clean up HDFSMetadataLog for Hadoop 2.6+

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866445#comment-15866445 ] Apache Spark commented on SPARK-19599: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Created] (SPARK-19600) ArrayIndexOutOfBoundsException in ALS

2017-02-14 Thread zhengxiang pan (JIRA)
zhengxiang pan created SPARK-19600: -- Summary: ArrayIndexOutOfBoundsException in ALS Key: SPARK-19600 URL: https://issues.apache.org/jira/browse/SPARK-19600 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-19529) TransportClientFactory.createClient() shouldn't call awaitUninterruptibly()

2017-02-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-19529. Resolution: Fixed > TransportClientFactory.createClient() shouldn't call awaitUninterruptibly() >

[jira] [Assigned] (SPARK-19453) Correct DataFrame.replace docs

2017-02-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-19453: --- Assignee: Maciej Szymkiewicz > Correct DataFrame.replace docs > -- > >

[jira] [Updated] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Armin Braun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Armin Braun updated SPARK-19592: Affects Version/s: 2.2.0 > Duplication in Test Configuration Relating to SparkConf Settings Should

[jira] [Assigned] (SPARK-19162) UserDefinedFunction constructor should verify that func is callable

2017-02-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-19162: --- Assignee: Maciej Szymkiewicz > UserDefinedFunction constructor should verify that func is callable

[jira] [Commented] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866311#comment-15866311 ] Sean Owen commented on SPARK-19592: --- What about tests that make their own conf or need to? I don't

[jira] [Commented] (SPARK-19598) Remove the alias parameter in UnresolvedRelation

2017-02-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866410#comment-15866410 ] Reynold Xin commented on SPARK-19598: - cc [~windpiger] are you interested in working on this? >

[jira] [Commented] (SPARK-14523) Feature parity for Statistics ML with MLlib

2017-02-14 Thread Timothy Hunter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866295#comment-15866295 ] Timothy Hunter commented on SPARK-14523: Also, the correlation is missing the multivariate case.

[jira] [Resolved] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-02-14 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19552. -- Resolution: Later > Upgrade Netty version to 4.1.8 final >

[jira] [Assigned] (SPARK-19587) Disallow when sort columns are part of partitioning columns

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19587: Assignee: Apache Spark > Disallow when sort columns are part of partitioning columns >

[jira] [Commented] (SPARK-19587) Disallow when sort columns are part of partitioning columns

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866437#comment-15866437 ] Apache Spark commented on SPARK-19587: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Updated] (SPARK-19529) TransportClientFactory.createClient() shouldn't call awaitUninterruptibly()

2017-02-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-19529: --- Target Version/s: 1.6.4, 2.0.3, 2.1.1, 2.2.0 (was: 1.6.3, 2.0.3, 2.1.1, 2.2.0) >

[jira] [Resolved] (SPARK-19162) UserDefinedFunction constructor should verify that func is callable

2017-02-14 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-19162. - Resolution: Fixed Fix Version/s: 2.2.0 > UserDefinedFunction constructor should verify that func

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2017-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866232#comment-15866232 ] Xiao Li commented on SPARK-13219: - If the predicates can be inferred, we definitely can push them down.

[jira] [Resolved] (SPARK-19582) DataFrameReader conceptually inadequate

2017-02-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19582. --- Resolution: Invalid I don't understand what this is describing. Is it a dependency conflict? if so,

[jira] [Assigned] (SPARK-19587) Disallow when sort columns are part of partitioning columns

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19587: Assignee: (was: Apache Spark) > Disallow when sort columns are part of partitioning

[jira] [Resolved] (SPARK-19163) Lazy creation of the _judf

2017-02-14 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-19163. Resolution: Fixed Fix Version/s: 2.2.0 > Lazy creation of the _judf >

[jira] [Resolved] (SPARK-19571) tests are failing to run on Windows with another instance Derby error with Hadoop 2.6.5

2017-02-14 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-19571. -- Resolution: Fixed Assignee: Hyukjin Kwon > tests are failing to run on Windows with

[jira] [Created] (SPARK-19598) Remove the alias parameter in UnresolvedRelation

2017-02-14 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-19598: --- Summary: Remove the alias parameter in UnresolvedRelation Key: SPARK-19598 URL: https://issues.apache.org/jira/browse/SPARK-19598 Project: Spark Issue Type:

[jira] [Created] (SPARK-19599) Clean up HDFSMetadataLog for Hadoop 2.6+

2017-02-14 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-19599: Summary: Clean up HDFSMetadataLog for Hadoop 2.6+ Key: SPARK-19599 URL: https://issues.apache.org/jira/browse/SPARK-19599 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19529) TransportClientFactory.createClient() shouldn't call awaitUninterruptibly()

2017-02-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-19529: --- Fix Version/s: 2.2.0 2.1.1 2.0.3 >

[jira] [Updated] (SPARK-19599) Clean up HDFSMetadataLog for Hadoop 2.6+

2017-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-19599: --- Affects Version/s: (was: 2.1.0) 2.2.0 > Clean up HDFSMetadataLog

[jira] [Commented] (SPARK-19518) IGNORE NULLS in first_value / last_value should be supported in SQL statements

2017-02-14 Thread Sameer Abhyankar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866461#comment-15866461 ] Sameer Abhyankar commented on SPARK-19518: -- I have come across this issue recently with Spark

[jira] [Resolved] (SPARK-19600) ArrayIndexOutOfBoundsException in ALS

2017-02-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19600. --- Resolution: Duplicate Questions go to the mailing list. Don't open duplicate JIRAs to re-ask. >

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865330#comment-15865330 ] Saisai Shao commented on SPARK-19579: - If you're using Python API to writing Streaming application,

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865360#comment-15865360 ] Saisai Shao commented on SPARK-19579: - Only this specific part is not supported, all the that Python

[jira] [Closed] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-19579. --- Resolution: Won't Fix > spark-submit fails to run Kafka Stream python script >

[jira] [Comment Edited] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865359#comment-15865359 ] xukun edited comment on SPARK-18113 at 2/14/17 8:34 AM: [~jinxing6...@126.com]

[jira] [Resolved] (SPARK-19586) Incorrect push down filter for double negative in SQL

2017-02-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-19586. -- Resolution: Fixed Fix Version/s: (was: 2.1.0) 2.0.3 >

[jira] [Commented] (SPARK-19586) Incorrect push down filter for double negative in SQL

2017-02-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865366#comment-15865366 ] Takeshi Yamamuro commented on SPARK-19586: -- This issue has been fixed in

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Piotr Nestorow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865329#comment-15865329 ] Piotr Nestorow commented on SPARK-19579: Are you positive that only Kafka 0.8 is supported. This

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865346#comment-15865346 ] Saisai Shao commented on SPARK-19579: - The support of Python API is rejected by the community, so

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Piotr Nestorow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865350#comment-15865350 ] Piotr Nestorow commented on SPARK-19579: Are you stating that the Apache Spark Python API support

[jira] [Commented] (SPARK-19587) Disallow when sort columns are part of partitioning columns

2017-02-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865371#comment-15865371 ] Takeshi Yamamuro commented on SPARK-19587: -- Do you work on this? If no, I'll do that, Thanks! >

[jira] [Comment Edited] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865360#comment-15865360 ] Saisai Shao edited comment on SPARK-19579 at 2/14/17 8:33 AM: -- Only this

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865359#comment-15865359 ] xukun commented on SPARK-18113: --- [~jinxing6...@126.com] yes. reproduce steps: 1. executor is running task

[jira] [Updated] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Armin Braun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Armin Braun updated SPARK-19592: Description: This configuration for Surefire, Scalatest is duplicated in the parent POM as well

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-14 Thread Piotr Nestorow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865343#comment-15865343 ] Piotr Nestorow commented on SPARK-19579: OK What's the roadmap for this? When this will be fixed?

[jira] [Created] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Armin Braun (JIRA)
Armin Braun created SPARK-19592: --- Summary: Duplication in Test Configuration Relating to SparkConf Settings Should be Removed Key: SPARK-19592 URL: https://issues.apache.org/jira/browse/SPARK-19592

[jira] [Updated] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Armin Braun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Armin Braun updated SPARK-19592: Description: This configuration for Surefire, Scalatest is duplicated in the parent POM as well

[jira] [Resolved] (SPARK-19589) Removal of SQLGEN files

2017-02-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-19589. --- Resolution: Fixed Fix Version/s: 2.2.0 > Removal of SQLGEN files >

[jira] [Assigned] (SPARK-19531) History server doesn't refresh jobs for long-life apps like thriftserver

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19531: Assignee: (was: Apache Spark) > History server doesn't refresh jobs for long-life

[jira] [Commented] (SPARK-19531) History server doesn't refresh jobs for long-life apps like thriftserver

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865589#comment-15865589 ] Apache Spark commented on SPARK-19531: -- User 'dosoft' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865644#comment-15865644 ] xukun edited comment on SPARK-18113 at 2/14/17 11:50 AM: - [~aash] Thanks for

[jira] [Comment Edited] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865644#comment-15865644 ] xukun edited comment on SPARK-18113 at 2/14/17 12:01 PM: - [~aash] Thanks for

[jira] [Commented] (SPARK-16475) Broadcast Hint for SQL Queries

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865688#comment-15865688 ] Apache Spark commented on SPARK-16475: -- User 'rxin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19531) History server doesn't refresh jobs for long-life apps like thriftserver

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19531: Assignee: Apache Spark > History server doesn't refresh jobs for long-life apps like

[jira] [Comment Edited] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865644#comment-15865644 ] xukun edited comment on SPARK-18113 at 2/14/17 11:58 AM: - [~aash] Thanks for

[jira] [Updated] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Armin Braun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Armin Braun updated SPARK-19592: Description: This configuration for Surefire, Scalatest is duplicated in the parent POM as well

[jira] [Created] (SPARK-19593) Records read per each kinesis transaction

2017-02-14 Thread Sarath Chandra Jiguru (JIRA)
Sarath Chandra Jiguru created SPARK-19593: - Summary: Records read per each kinesis transaction Key: SPARK-19593 URL: https://issues.apache.org/jira/browse/SPARK-19593 Project: Spark

[jira] [Commented] (SPARK-10057) Faill to load class org.slf4j.impl.StaticLoggerBinder

2017-02-14 Thread Samir Ouldsaadi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865601#comment-15865601 ] Samir Ouldsaadi commented on SPARK-10057: - Same issue happening in spark 2.1.0 to reproduce:

[jira] [Comment Edited] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865644#comment-15865644 ] xukun edited comment on SPARK-18113 at 2/14/17 11:51 AM: - [~aash] Thanks for

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865644#comment-15865644 ] xukun commented on SPARK-18113: --- [~aash] Thanks for your reply. I think

[jira] [Comment Edited] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread xukun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865644#comment-15865644 ] xukun edited comment on SPARK-18113 at 2/14/17 11:50 AM: - [~aash] Thanks for

[jira] [Created] (SPARK-19594) StreamingQueryListener fails to handle QueryTerminatedEvent if more then one listeners exists

2017-02-14 Thread Eyal Zituny (JIRA)
Eyal Zituny created SPARK-19594: --- Summary: StreamingQueryListener fails to handle QueryTerminatedEvent if more then one listeners exists Key: SPARK-19594 URL: https://issues.apache.org/jira/browse/SPARK-19594

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865536#comment-15865536 ] Andrew Ash commented on SPARK-18113: Thanks for the updates you both. I've been working with a

[jira] [Commented] (SPARK-19588) Allow putting keytab file to HDFS location specified in spark.yarn.keytab

2017-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865434#comment-15865434 ] Saisai Shao commented on SPARK-19588: - Putting on HDFS still requires downloading to local disk for

[jira] [Updated] (SPARK-19593) Records read per each kinesis transaction

2017-02-14 Thread Sarath Chandra Jiguru (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Chandra Jiguru updated SPARK-19593: -- Description: The question is related to spark streaming+kinesis integration

[jira] [Updated] (SPARK-19582) DataFrameReader conceptually inadequate

2017-02-14 Thread James Q. Arnold (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Q. Arnold updated SPARK-19582: Description: DataFrameReader assumes it "understands" all data sources (local file system,

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

2017-02-14 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865778#comment-15865778 ] Andrew Ash commented on SPARK-18113: [~xukun] the scenario you describe should be accommodated by the

[jira] [Updated] (SPARK-19595) from_json produces only a single row when input is a json array

2017-02-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19595: - Description: Currently, {{from_json}} reads a single row when it is a json array. For example,

[jira] [Created] (SPARK-19595) from_json produces only a single row when input is a json array

2017-02-14 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-19595: Summary: from_json produces only a single row when input is a json array Key: SPARK-19595 URL: https://issues.apache.org/jira/browse/SPARK-19595 Project: Spark

[jira] [Commented] (SPARK-19596) After a Stage is completed, all Tasksets for the stage should be marked as zombie

2017-02-14 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865971#comment-15865971 ] Imran Rashid commented on SPARK-19596: -- cc [~kayousterhout] [~jinxing6...@126.com] [~markhamstra] >

[jira] [Created] (SPARK-19596) After a Stage is completed, all Tasksets for the stage should be marked as zombie

2017-02-14 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-19596: Summary: After a Stage is completed, all Tasksets for the stage should be marked as zombie Key: SPARK-19596 URL: https://issues.apache.org/jira/browse/SPARK-19596

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2017-02-14 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866210#comment-15866210 ] Nick Dimiduk commented on SPARK-13219: -- bq. The result is that no matter the number of tables and

[jira] [Closed] (SPARK-19166) change method name from InsertIntoHadoopFsRelationCommand.deleteMatchingPartitions to InsertIntoHadoopFsRelationCommand.deleteMatchingPrefix

2017-02-14 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Song Jun closed SPARK-19166. Resolution: Not A Bug minor issue > change method name from >

[jira] [Assigned] (SPARK-19595) from_json produces only a single row when input is a json array

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19595: Assignee: Apache Spark > from_json produces only a single row when input is a json array

[jira] [Assigned] (SPARK-19595) from_json produces only a single row when input is a json array

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19595: Assignee: (was: Apache Spark) > from_json produces only a single row when input is a

[jira] [Updated] (SPARK-19597) ExecutorSuite should have test for tasks that are not deserialiazable

2017-02-14 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-19597: - Description: We should have a test case that ensures that Executors gracefully handle a task

[jira] [Created] (SPARK-19597) ExecutorSuite should have test for tasks that are not deserialiazable

2017-02-14 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-19597: Summary: ExecutorSuite should have test for tasks that are not deserialiazable Key: SPARK-19597 URL: https://issues.apache.org/jira/browse/SPARK-19597 Project: Spark

[jira] [Commented] (SPARK-19586) Incorrect push down filter for double negative in SQL

2017-02-14 Thread Everett Anderson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866177#comment-15866177 ] Everett Anderson commented on SPARK-19586: -- [~maropu] Awesome! Thanks so much! > Incorrect push

[jira] [Commented] (SPARK-18699) Spark CSV parsing types other than String throws exception when malformed

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865959#comment-15865959 ] Apache Spark commented on SPARK-18699: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-19595) from_json produces only a single row when input is a json array

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866089#comment-15866089 ] Apache Spark commented on SPARK-19595: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-19597) ExecutorSuite should have test for tasks that are not deserialiazable

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19597: Assignee: Apache Spark (was: Imran Rashid) > ExecutorSuite should have test for tasks

[jira] [Commented] (SPARK-19597) ExecutorSuite should have test for tasks that are not deserialiazable

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866158#comment-15866158 ] Apache Spark commented on SPARK-19597: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19597) ExecutorSuite should have test for tasks that are not deserialiazable

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19597: Assignee: Imran Rashid (was: Apache Spark) > ExecutorSuite should have test for tasks

[jira] [Commented] (SPARK-19038) Can't find keytab file when using Hive catalog

2017-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865415#comment-15865415 ] Apache Spark commented on SPARK-19038: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Commented] (SPARK-18541) Add pyspark.sql.Column.aliasWithMetadata to allow dynamic metadata management in pyspark SQL API

2017-02-14 Thread Shea Parkes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866393#comment-15866393 ] Shea Parkes commented on SPARK-18541: - Thank you very much! > Add

[jira] [Commented] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Armin Braun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866428#comment-15866428 ] Armin Braun commented on SPARK-19592: - Imo this also relates to the ability to handle

[jira] [Commented] (SPARK-19592) Duplication in Test Configuration Relating to SparkConf Settings Should be Removed

2017-02-14 Thread Armin Braun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866367#comment-15866367 ] Armin Braun commented on SPARK-19592: - [~srowen] {quote} What about tests that make their own conf

[jira] [Updated] (SPARK-19529) TransportClientFactory.createClient() shouldn't call awaitUninterruptibly()

2017-02-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-19529: --- Fix Version/s: 1.6.4 > TransportClientFactory.createClient() shouldn't call awaitUninterruptibly() >

[jira] [Commented] (SPARK-19600) ArrayIndexOutOfBoundsException in ALS

2017-02-14 Thread zhengxiang pan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866516#comment-15866516 ] zhengxiang pan commented on SPARK-19600: are you sure it is duplicated issue as SPARK-3080 even

[jira] [Commented] (SPARK-19208) MultivariateOnlineSummarizer performance optimization

2017-02-14 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866755#comment-15866755 ] Nick Pentreath commented on SPARK-19208: Ah right I see - yes rewrite rules would be a good

[jira] [Updated] (SPARK-19593) Records read per each kinesis transaction

2017-02-14 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19593: - Component/s: (was: Structured Streaming) (was: Spark Core)

[jira] [Commented] (SPARK-19208) MultivariateOnlineSummarizer performance optimization

2017-02-14 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866675#comment-15866675 ] Nick Pentreath commented on SPARK-19208: When I said "estimator-like", I didn't mean it should

  1   2   >