[jira] [Created] (SPARK-25538) incorrect row counts after distinct()

2018-09-25 Thread Steven Rand (JIRA)
Steven Rand created SPARK-25538: --- Summary: incorrect row counts after distinct() Key: SPARK-25538 URL: https://issues.apache.org/jira/browse/SPARK-25538 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25537) spark.pyspark.driver.python when set in code doesnt work

2018-09-25 Thread Venkat Sambath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkat Sambath updated SPARK-25537: --- Description: spark.pyspark.driver.python, spark.pyspark.python when set in code doesnt get

[jira] [Commented] (SPARK-25536) executorSource.METRIC read wrong record in Executor.scala Line444

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628267#comment-16628267 ] shahid commented on SPARK-25536: I will raise a pr > executorSource.METRIC read wrong record in

[jira] [Created] (SPARK-25537) spark.pyspark.driver.python when set in code doesnt work

2018-09-25 Thread Venkat Sambath (JIRA)
Venkat Sambath created SPARK-25537: -- Summary: spark.pyspark.driver.python when set in code doesnt work Key: SPARK-25537 URL: https://issues.apache.org/jira/browse/SPARK-25537 Project: Spark

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628231#comment-16628231 ] Wenchen Fan commented on SPARK-25378: - [~mengxr] what do you think? This is not a real compatibility

[jira] [Created] (SPARK-25536) executorSource.METRIC read wrong record in Executor.scala Line444

2018-09-25 Thread ZhuoerXu (JIRA)
ZhuoerXu created SPARK-25536: Summary: executorSource.METRIC read wrong record in Executor.scala Line444 Key: SPARK-25536 URL: https://issues.apache.org/jira/browse/SPARK-25536 Project: Spark

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-25 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628225#comment-16628225 ] Liang-Chi Hsieh commented on SPARK-25378: - Don't we have any decision on this yet? >

[jira] [Updated] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-25 Thread huanghuai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huanghuai updated SPARK-25523: -- Description: public static void test2() throws Exception{ String ckUrlPrefix="jdbc:clickhouse://";

[jira] [Updated] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-25 Thread huanghuai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huanghuai updated SPARK-25523: -- Description: public static void test2() throws Exception{ String ckUrlPrefix="jdbc:clickhouse://";

[jira] [Commented] (SPARK-23839) consider bucket join in cost-based JoinReorder rule

2018-09-25 Thread Xiaoju Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628184#comment-16628184 ] Xiaoju Wu commented on SPARK-23839: --- [~smilegator] Is there any plan on the cost-based optimizer? >

[jira] [Updated] (SPARK-25517) Spark DataFrame option inferSchema="true", dataFormat=MM/dd/yyyy, fails to detect date type from the csv file while reading

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25517: -- Environment: (was: Spark 2.3.0) > Spark DataFrame option inferSchema="true",

[jira] [Resolved] (SPARK-25517) Spark DataFrame option inferSchema="true", dataFormat=MM/dd/yyyy, fails to detect date type from the csv file while reading

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25517. --- Resolution: Duplicate According to the comments on the PR, I'll close this as `Duplicate`

[jira] [Closed] (SPARK-25518) Spark kafka delegation token supported

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25518. - > Spark kafka delegation token supported > --- > >

[jira] [Closed] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25526. - > SparkSQL over REST API not JDBC > --- > > Key:

[jira] [Updated] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25526: -- Component/s: (was: Spark Submit) (was: Spark Core)

[jira] [Resolved] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25526. --- Resolution: Won't Fix > SparkSQL over REST API not JDBC > --- >

[jira] [Commented] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628165#comment-16628165 ] Dongjoon Hyun commented on SPARK-25526: --- In addition to that, recently LIVY-494 starts to support

[jira] [Commented] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628161#comment-16628161 ] Dongjoon Hyun commented on SPARK-25526: --- Could you try Apache Livy? Apache Livy supports all

[jira] [Closed] (SPARK-25511) Map with "null" key not working in spark 2.3

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25511. - > Map with "null" key not working in spark 2.3 > > >

[jira] [Updated] (SPARK-25511) Map with "null" key not working in spark 2.3

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25511: -- Component/s: (was: Optimizer) SQL > Map with "null" key not working in

[jira] [Resolved] (SPARK-25511) Map with "null" key not working in spark 2.3

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25511. --- Resolution: Invalid Hi, [~ravi_b_shankar]. Historically, it's designed and documented like

[jira] [Comment Edited] (SPARK-23985) predicate push down doesn't work with simple compound partition spec

2018-09-25 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625876#comment-16625876 ] Yuming Wang edited comment on SPARK-23985 at 9/26/18 1:54 AM: --  [~uzadude]

[jira] [Resolved] (SPARK-25514) Generating pretty JSON by to_json

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25514. -- Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22534

[jira] [Assigned] (SPARK-25514) Generating pretty JSON by to_json

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25514: Assignee: Maxim Gekk > Generating pretty JSON by to_json >

[jira] [Resolved] (SPARK-21291) R bucketBy partitionBy API

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21291. -- Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22537

[jira] [Assigned] (SPARK-21291) R bucketBy partitionBy API

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-21291: Assignee: Huaxin Gao > R bucketBy partitionBy API > -- > >

[jira] [Issue Comment Deleted] (SPARK-25534) Make `SQLHelper` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Comment: was deleted (was: User 'dongjoon-hyun' has created a pull request for this issue:

[jira] [Updated] (SPARK-25534) Make `SQLHelper` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Summary: Make `SQLHelper` trait (was: Make `SupportWithSQLConf` trait) > Make `SQLHelper`

[jira] [Updated] (SPARK-25534) Make `SQLHelper` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 7 `withTempPath` and 6 `withSQLConf` functions. This PR

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628120#comment-16628120 ] Apache Spark commented on SPARK-25501: -- User 'merlintang' has created a pull request for this

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628119#comment-16628119 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] if you like, you can propose a PR based my PR

[jira] [Assigned] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25501: Assignee: (was: Apache Spark) > Kafka delegation token support >

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628117#comment-16628117 ] Apache Spark commented on SPARK-25501: -- User 'merlintang' has created a pull request for this

[jira] [Assigned] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25501: Assignee: Apache Spark > Kafka delegation token support > --

[jira] [Resolved] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25422. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22546

[jira] [Assigned] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25422: --- Assignee: Imran Rashid > flaky test: org.apache.spark.DistributedSuite.caching on disk,

[jira] [Commented] (SPARK-23839) consider bucket join in cost-based JoinReorder rule

2018-09-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628080#comment-16628080 ] Xiao Li commented on SPARK-23839: - To implement CBO in planner, we need a major change in our planner.

[jira] [Commented] (SPARK-25535) Work around bad error checking in commons-crypto

2018-09-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628078#comment-16628078 ] Marcelo Vanzin commented on SPARK-25535: I filed CRYPTO-141 for the commons-crypto fix. But in

[jira] [Created] (SPARK-25535) Work around bad error checking in commons-crypto

2018-09-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25535: -- Summary: Work around bad error checking in commons-crypto Key: SPARK-25535 URL: https://issues.apache.org/jira/browse/SPARK-25535 Project: Spark Issue

[jira] [Updated] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Description: Test steps:  1) bin/spark-shell {code:java} sc.parallelize(1 to 5, 5).collect()

[jira] [Updated] (SPARK-25102) Write Spark version information to Parquet file footers

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25102: -- Component/s: (was: Spark Core) SQL > Write Spark version information to

[jira] [Updated] (SPARK-25102) Write Spark version information to Parquet file footers

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25102: -- Priority: Minor (was: Major) > Write Spark version information to Parquet file footers >

[jira] [Commented] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627979#comment-16627979 ] Apache Spark commented on SPARK-25533: -- User 'shahidki31' has created a pull request for this

[jira] [Assigned] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25533: Assignee: (was: Apache Spark) > Inconsistent message for Completed Jobs in the

[jira] [Assigned] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25533: Assignee: Apache Spark > Inconsistent message for Completed Jobs in the JobUI, when

[jira] [Commented] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627974#comment-16627974 ] Apache Spark commented on SPARK-25534: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25534: Assignee: Dongjoon Hyun (was: Apache Spark) > Make `SupportWithSQLConf` trait >

[jira] [Assigned] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25534: Assignee: Apache Spark (was: Dongjoon Hyun) > Make `SupportWithSQLConf` trait >

[jira] [Commented] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627973#comment-16627973 ] Apache Spark commented on SPARK-25534: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 6 `withSQLConf` function. This issue aims to remove

[jira] [Updated] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 6 `withSQLConf` function. This issue aims to remove

[jira] [Updated] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 6 `withSQLConf` function. This issue aims to remove

[jira] [Commented] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627961#comment-16627961 ] Dongjoon Hyun commented on SPARK-25534: --- I'll make a PR soon. > Make `SupportWithSQLConf` trait >

[jira] [Assigned] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25534: - Assignee: Dongjoon Hyun > Make `SupportWithSQLConf` trait >

[jira] [Created] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-25534: - Summary: Make `SupportWithSQLConf` trait Key: SPARK-25534 URL: https://issues.apache.org/jira/browse/SPARK-25534 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627934#comment-16627934 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] any updates on your PR, I can propose a PR as

[jira] [Updated] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Summary: Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: (was: Screenshot from 2018-09-26 00-42-00.png) > Inconsistent message of Completed Jobs in

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Description: Test steps:  1) bin/spark-shell {code:java} sc.parallelize(1 to 5, 5).collect()

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: Screenshot from 2018-09-26 00-46-35.png > Inconsistent message of Completed Jobs in the JobUI,

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: Screenshot from 2018-09-26 00-42-00.png > Inconsistent message of Completed Jobs in the JobUI,

[jira] [Commented] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627823#comment-16627823 ] shahid commented on SPARK-25533: I am working on it. > Inconsistent message of Completed Jobs in the

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: Screenshot from 2018-09-26 00-42-00.png > Inconsistent message of Completed Jobs in the JobUI,

[jira] [Created] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
shahid created SPARK-25533: -- Summary: Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2 Key: SPARK-25533 URL: https://issues.apache.org/jira/browse/SPARK-25533

[jira] [Resolved] (SPARK-25495) FetchedData.reset doesn't reset _nextOffsetInFetchedData and _offsetAfterPoll

2018-09-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25495. -- Resolution: Fixed Fix Version/s: 2.4.0 > FetchedData.reset doesn't reset

[jira] [Assigned] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-09-25 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-17636: --- Assignee: DB Tsai > Parquet filter push down doesn't handle struct fields >

[jira] [Updated] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-09-25 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-17636: Fix Version/s: 2.5.0 > Parquet filter push down doesn't handle struct fields >

[jira] [Resolved] (SPARK-25486) Refactor SortBenchmark to use main method

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25486. --- Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22495

[jira] [Assigned] (SPARK-25486) Refactor SortBenchmark to use main method

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25486: - Assignee: yucai > Refactor SortBenchmark to use main method >

[jira] [Commented] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627650#comment-16627650 ] Apache Spark commented on SPARK-25528: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627649#comment-16627649 ] Apache Spark commented on SPARK-25528: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25528: Assignee: Wenchen Fan (was: Apache Spark) > data source V2 read side API refactoring >

[jira] [Commented] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627648#comment-16627648 ] Apache Spark commented on SPARK-25528: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25528: Assignee: Apache Spark (was: Wenchen Fan) > data source V2 read side API refactoring >

[jira] [Resolved] (SPARK-23521) SPIP: Standardize SQL logical plans with DataSourceV2

2018-09-25 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved SPARK-23521. --- Resolution: Fixed Marking this as "FIxed" because the vote passed. > SPIP: Standardize SQL logical

[jira] [Commented] (SPARK-25527) Job stuck waiting for last stage to start

2018-09-25 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627593#comment-16627593 ] Li Yuanjian commented on SPARK-25527: - {quote} There are no Tasks waiting for completion, and the

[jira] [Created] (SPARK-25532) A stable and efficient row representation

2018-09-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25532: --- Summary: A stable and efficient row representation Key: SPARK-25532 URL: https://issues.apache.org/jira/browse/SPARK-25532 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-25531) new write APIs for data source v2

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627572#comment-16627572 ] Wenchen Fan commented on SPARK-25531: - Hi [~rdblue], I've created this umbrella JIRA ticket to track

[jira] [Updated] (SPARK-23521) SPIP: Standardize SQL logical plans with DataSourceV2

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23521: Issue Type: Sub-task (was: Improvement) Parent: SPARK-25531 > SPIP: Standardize SQL

[jira] [Updated] (SPARK-24923) DataSourceV2: Add CTAS and RTAS logical operations

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24923: Parent Issue: SPARK-25531 (was: SPARK-22386) > DataSourceV2: Add CTAS and RTAS logical

[jira] [Updated] (SPARK-24253) DataSourceV2: Add DeleteSupport for delete and overwrite operations

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24253: Parent Issue: SPARK-25531 (was: SPARK-22386) > DataSourceV2: Add DeleteSupport for delete and

[jira] [Updated] (SPARK-24251) DataSourceV2: Add AppendData logical operation

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24251: Parent Issue: SPARK-25531 (was: SPARK-22386) > DataSourceV2: Add AppendData logical operation >

[jira] [Created] (SPARK-25531) new write APIs for data source v2

2018-09-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25531: --- Summary: new write APIs for data source v2 Key: SPARK-25531 URL: https://issues.apache.org/jira/browse/SPARK-25531 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-25 Thread huanghuai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huanghuai updated SPARK-25523: -- Summary: Multi thread execute sparkSession.read().jdbc(url, table, properties) problem (was:

[jira] [Resolved] (SPARK-25189) CustomMetrics should be created with ScanConfig

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25189. - Resolution: Not A Problem The {{CustomMetrics}} has been reverted > CustomMetrics should be

[jira] [Updated] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25528: Description: refactor the read side API according to this abstraction {code} batch: catalog ->

[jira] [Created] (SPARK-25530) data source v2 write side API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25530: --- Summary: data source v2 write side API refactoring Key: SPARK-25530 URL: https://issues.apache.org/jira/browse/SPARK-25530 Project: Spark Issue Type: Sub-task

[jira] [Deleted] (SPARK-25529) data source V2 read side API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan deleted SPARK-25529: > data source V2 read side API refactoring > > >

[jira] [Created] (SPARK-25529) data source V2 read side API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25529: --- Summary: data source V2 read side API refactoring Key: SPARK-25529 URL: https://issues.apache.org/jira/browse/SPARK-25529 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25528: --- Summary: data source V2 read side API refactoring Key: SPARK-25528 URL: https://issues.apache.org/jira/browse/SPARK-25528 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-25390) data source V2 API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25390: Target Version/s: 2.5.0 > data source V2 API refactoring > -- > >

[jira] [Updated] (SPARK-25390) data source V2 API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25390: Description: Currently it's not very clear how we should abstract data source v2 API. The

[jira] [Updated] (SPARK-25390) data source V2 API refactoring

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25390: Summary: data source V2 API refactoring (was: finalize the abstraction of data source V2 API) >

[jira] [Updated] (SPARK-25390) finalize the abstraction of data source V2 API

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25390: Affects Version/s: (was: 3.0.0) 2.5.0 > finalize the abstraction of

[jira] [Resolved] (SPARK-25500) Specify configmap and secrets in Spark driver and executor pods in Kubernetes

2018-09-25 Thread Abhishek Rao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Rao resolved SPARK-25500. -- Resolution: Won't Fix As mentioned in ticket, we'll use the templates way to mount configmap.

[jira] [Commented] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627466#comment-16627466 ] Apache Spark commented on SPARK-25422: -- User 'squito' has created a pull request for this issue:

[jira] [Commented] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627464#comment-16627464 ] Apache Spark commented on SPARK-25422: -- User 'squito' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-23985) predicate push down doesn't work with simple compound partition spec

2018-09-25 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625876#comment-16625876 ] Yuming Wang edited comment on SPARK-23985 at 9/25/18 2:19 PM: --  [~uzadude]

[jira] [Commented] (SPARK-24064) [Spark SQL] Create table using csv does not support binary column Type

2018-09-25 Thread Gururaj Shetty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627376#comment-16627376 ] Gururaj Shetty commented on SPARK-24064: If Binary is not supported only for CSV, then the same

[jira] [Created] (SPARK-25527) Job stuck waiting for last stage to start

2018-09-25 Thread Ran Haim (JIRA)
Ran Haim created SPARK-25527: Summary: Job stuck waiting for last stage to start Key: SPARK-25527 URL: https://issues.apache.org/jira/browse/SPARK-25527 Project: Spark Issue Type: Bug

  1   2   >