[jira] [Commented] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-15 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818658#comment-16818658 ] shahid commented on SPARK-27465: I will analyze the issue. > Kafka Client 0.11.0.0 is not Supporting

[jira] [Commented] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-15 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818662#comment-16818662 ] shahid commented on SPARK-27468: I would like to analyze the issue. > "Storage Level" in "RDD Storage

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-15 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818649#comment-16818649 ] Bryan Cutler commented on SPARK-27396: -- Thanks for this [~revans2], overall I think the proposal

[jira] [Commented] (SPARK-25348) Data source for binary files

2019-04-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818640#comment-16818640 ] Xiangrui Meng commented on SPARK-25348: --- I created two follow-up tasks: * DocumentationL

[jira] [Comment Edited] (SPARK-25348) Data source for binary files

2019-04-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818640#comment-16818640 ] Xiangrui Meng edited comment on SPARK-25348 at 4/16/19 5:19 AM: I

[jira] [Created] (SPARK-27473) Support filter push down for status fields in binary file data source

2019-04-15 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27473: - Summary: Support filter push down for status fields in binary file data source Key: SPARK-27473 URL: https://issues.apache.org/jira/browse/SPARK-27473 Project:

[jira] [Updated] (SPARK-25348) Data source for binary files

2019-04-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25348: -- Component/s: (was: ML) > Data source for binary files > > >

[jira] [Created] (SPARK-27472) Docuement binary file data source in Spark user guide

2019-04-15 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27472: - Summary: Docuement binary file data source in Spark user guide Key: SPARK-27472 URL: https://issues.apache.org/jira/browse/SPARK-27472 Project: Spark

[jira] [Updated] (SPARK-25348) Data source for binary files

2019-04-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25348: -- Description: It would be useful to have a data source implementation for binary files, which

[jira] [Updated] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-15 Thread Praveen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Praveen updated SPARK-27465: Affects Version/s: 2.3.0 2.3.1 2.3.2

[jira] [Updated] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-15 Thread Praveen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Praveen updated SPARK-27465: Priority: Critical (was: Major) > Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package >

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2019-04-15 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818598#comment-16818598 ] Kevin Zhang commented on SPARK-24630: - thanks [~Jackey Lee] So I'm wondering what's blocking the pr

[jira] [Resolved] (SPARK-27436) Add spark.sql.optimizer.nonExcludedRules

2019-04-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27436. --- Resolution: Won't Do > Add spark.sql.optimizer.nonExcludedRules >

[jira] [Created] (SPARK-27471) Reorganize public v2 catalog API

2019-04-15 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-27471: - Summary: Reorganize public v2 catalog API Key: SPARK-27471 URL: https://issues.apache.org/jira/browse/SPARK-27471 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27386) Improve partition transform parsing

2019-04-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818457#comment-16818457 ] Reynold Xin commented on SPARK-27386: - [~rdblue] when will you fix this? > Improve partition

[jira] [Resolved] (SPARK-27351) Wrong outputRows estimation after AggregateEstimation with only null value column

2019-04-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27351. --- Resolution: Fixed Assignee: peng bo Fix Version/s: 3.0.0

[jira] [Updated] (SPARK-27452) Update zstd-jni to 1.3.8-9

2019-04-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27452: -- Summary: Update zstd-jni to 1.3.8-9 (was: Update zstd-jni to 1.3.8-7) > Update zstd-jni to

[jira] [Assigned] (SPARK-27452) Update zstd-jni to 1.3.8-9

2019-04-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27452: - Assignee: Dongjoon Hyun > Update zstd-jni to 1.3.8-9 > -- > >

[jira] [Created] (SPARK-27470) Upgrade pyrolite to 4.23

2019-04-15 Thread Sean Owen (JIRA)
Sean Owen created SPARK-27470: - Summary: Upgrade pyrolite to 4.23 Key: SPARK-27470 URL: https://issues.apache.org/jira/browse/SPARK-27470 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-24717) Split out min retain version of state for memory in HDFSBackedStateStoreProvider

2019-04-15 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818326#comment-16818326 ] Stavros Kontopoulos commented on SPARK-24717: - [~tdas] What is the point of having

[jira] [Updated] (SPARK-27463) SPIP: Support Dataframe Cogroup via Pandas UDFs

2019-04-15 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Martin updated SPARK-27463: - Description: Recent work on Pandas UDFs in Spark, has allowed for improved interoperability

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

2019-04-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818318#comment-16818318 ] Imran Rashid commented on SPARK-25250: -- I agree about opening a new jira. Wenchen discussed

[jira] [Assigned] (SPARK-27454) Spark image datasource fail when encounter some illegal images

2019-04-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27454: - Assignee: Weichen Xu > Spark image datasource fail when encounter some illegal images

[jira] [Resolved] (SPARK-27454) Spark image datasource fail when encounter some illegal images

2019-04-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-27454. --- Resolution: Fixed Fix Version/s: 3.0.0 > Spark image datasource fail when encounter

[jira] [Created] (SPARK-27469) Update Commons BeanUtils to 1.9.3

2019-04-15 Thread Sean Owen (JIRA)
Sean Owen created SPARK-27469: - Summary: Update Commons BeanUtils to 1.9.3 Key: SPARK-27469 URL: https://issues.apache.org/jira/browse/SPARK-27469 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818197#comment-16818197 ] Xiao Li commented on SPARK-27468: - cc [~Gengliang.Wang] > "Storage Level" in "RDD Storage Page" is not

[jira] [Created] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

2019-04-15 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-27468: Summary: "Storage Level" in "RDD Storage Page" is not correct Key: SPARK-27468 URL: https://issues.apache.org/jira/browse/SPARK-27468 Project: Spark Issue

[jira] [Updated] (SPARK-27458) Remind developer using IntelliJ to update maven version

2019-04-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27458: -- Priority: Minor (was: Major) > Remind developer using IntelliJ to update maven version >

[jira] [Updated] (SPARK-27467) Upgrade Maven to 3.6.1

2019-04-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27467: -- Description: This issue aim to upgrade Maven to 3.6.1 to bring JDK9+ patches like MNG-6506.

[jira] [Updated] (SPARK-27467) Upgrade Maven to 3.6.1

2019-04-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27467: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-24417 > Upgrade Maven to 3.6.1

[jira] [Updated] (SPARK-27430) broadcast hint should be respected for broadcast nested loop join

2019-04-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27430: Summary: broadcast hint should be respected for broadcast nested loop join (was:

[jira] [Created] (SPARK-27467) Upgrade Maven to 3.6.1

2019-04-15 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-27467: - Summary: Upgrade Maven to 3.6.1 Key: SPARK-27467 URL: https://issues.apache.org/jira/browse/SPARK-27467 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-27062) CatalogImpl.refreshTable should register query in cache with received tableName

2019-04-15 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong resolved SPARK-27062. -- Resolution: Duplicate > CatalogImpl.refreshTable should register query in cache with received

[jira] [Commented] (SPARK-27458) Remind developer using IntelliJ to update maven version

2019-04-15 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818100#comment-16818100 ] William Wong commented on SPARK-27458: -- PR ([https://github.com/apache/spark-website/pull/195]) was

[jira] [Updated] (SPARK-27466) LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark

2019-04-15 Thread Zoltan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan updated SPARK-27466: --- Environment: Spark version 2.2.0.2.6.4.92-2 Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM,

[jira] [Created] (SPARK-27466) LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark

2019-04-15 Thread Zoltan (JIRA)
Zoltan created SPARK-27466: -- Summary: LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark Key: SPARK-27466 URL: https://issues.apache.org/jira/browse/SPARK-27466

[jira] [Comment Edited] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

2019-04-15 Thread Prabhjot Singh Bharaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818070#comment-16818070 ] Prabhjot Singh Bharaj edited comment on SPARK-27409 at 4/15/19 3:30 PM:

[jira] [Commented] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

2019-04-15 Thread Prabhjot Singh Bharaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818070#comment-16818070 ] Prabhjot Singh Bharaj commented on SPARK-27409: --- >  "kafka.ssl.keystore.location",

[jira] [Commented] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

2019-04-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818056#comment-16818056 ] Gabor Somogyi commented on SPARK-27409: --- Are you really sure you've followed the following

[jira] [Created] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-15 Thread Praveen (JIRA)
Praveen created SPARK-27465: --- Summary: Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package Key: SPARK-27465 URL: https://issues.apache.org/jira/browse/SPARK-27465 Project: Spark

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

2019-04-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818053#comment-16818053 ] Thomas Graves commented on SPARK-25250: --- [~cloud_fan] can you please add details as to where and

[jira] [Commented] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

2019-04-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818051#comment-16818051 ] Gabor Somogyi commented on SPARK-27409: --- Don't know a couple of things in you code: * Why do you

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-15 Thread Robert Joseph Evans (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817990#comment-16817990 ] Robert Joseph Evans commented on SPARK-27396: - There are actually a few public facing APIs I 

[jira] [Assigned] (SPARK-27459) Revise the exception message of schema inference failure in file source V2

2019-04-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27459: --- Assignee: Gengliang Wang > Revise the exception message of schema inference failure in

[jira] [Resolved] (SPARK-27459) Revise the exception message of schema inference failure in file source V2

2019-04-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27459. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24369

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817926#comment-16817926 ] Gabor Somogyi commented on SPARK-27330: --- Cool, ping me if you need review... > ForeachWriter is

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-15 Thread Eyal Zituny (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817918#comment-16817918 ] Eyal Zituny commented on SPARK-27330: - [~gsomogyi] yes, almost done with it > ForeachWriter is not

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817902#comment-16817902 ] Gabor Somogyi commented on SPARK-27330: --- [~eyalzit] are you working on this? Happy to file a PR if

[jira] [Created] (SPARK-27464) Add Constant instead of referring string literal used from many places

2019-04-15 Thread Shivu Sondur (JIRA)
Shivu Sondur created SPARK-27464: Summary: Add Constant instead of referring string literal used from many places Key: SPARK-27464 URL: https://issues.apache.org/jira/browse/SPARK-27464 Project:

[jira] [Updated] (SPARK-27463) SPIP: Support Dataframe Cogroup via Pandas UDFs

2019-04-15 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Martin updated SPARK-27463: - Description: h2. *Background and Motivation* Recently there has been a great deal of work in

[jira] [Created] (SPARK-27463) SPIP: Support Dataframe Cogroup via Pandas UDFs

2019-04-15 Thread Chris Martin (JIRA)
Chris Martin created SPARK-27463: Summary: SPIP: Support Dataframe Cogroup via Pandas UDFs Key: SPARK-27463 URL: https://issues.apache.org/jira/browse/SPARK-27463 Project: Spark Issue Type:

[jira] [Created] (SPARK-27462) Spark hive can not choose some columns in target table flexibly, when running insert into.

2019-04-15 Thread jiaan.geng (JIRA)
jiaan.geng created SPARK-27462: -- Summary: Spark hive can not choose some columns in target table flexibly, when running insert into. Key: SPARK-27462 URL: https://issues.apache.org/jira/browse/SPARK-27462

[jira] [Updated] (SPARK-27461) Not throwing error for Datatype mismatch

2019-04-15 Thread Mahasubramanian Maharajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahasubramanian Maharajan updated SPARK-27461: -- Priority: Critical (was: Major) > Not throwing error for Datatype

[jira] [Created] (SPARK-27461) Not throwing error for Datatype mismatch

2019-04-15 Thread Mahasubramanian Maharajan (JIRA)
Mahasubramanian Maharajan created SPARK-27461: - Summary: Not throwing error for Datatype mismatch Key: SPARK-27461 URL: https://issues.apache.org/jira/browse/SPARK-27461 Project: Spark

[jira] [Resolved] (SPARK-27173) For hive parquet table,codes(lz4,brotli,zstd) are not available

2019-04-15 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian resolved SPARK-27173. - Resolution: Won't Fix > For hive parquet table,codes(lz4,brotli,zstd) are not available >