[jira] [Resolved] (SPARK-27517) python.PythonRDD: Error while sending iterator

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27517. -- Resolution: Incomplete > python.PythonRDD: Error while sending iterator >

[jira] [Commented] (SPARK-27517) python.PythonRDD: Error while sending iterator

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821843#comment-16821843 ] Hyukjin Kwon commented on SPARK-27517: -- Please just don't copy and paste the error message. No one

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821845#comment-16821845 ] Hyukjin Kwon commented on SPARK-27519: -- I can run if you post it in higher versions of Spark. >

[jira] [Resolved] (SPARK-27516) java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27516. -- Resolution: Incomplete > java.util.concurrent.TimeoutException: Futures timed out after

[jira] [Commented] (SPARK-27516) java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821844#comment-16821844 ] Hyukjin Kwon commented on SPARK-27516: -- I can't reproduce this. I suspect this environment specific

[jira] [Commented] (SPARK-27429) [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821763#comment-16821763 ] Liang-Chi Hsieh commented on SPARK-27429: - Generally I think you can always know which are the

[jira] [Commented] (SPARK-27511) Spark Streaming Driver Memory

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821847#comment-16821847 ] Hyukjin Kwon commented on SPARK-27511: -- Let's ask questions into mailing lists rather then filing

[jira] [Resolved] (SPARK-27511) Spark Streaming Driver Memory

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27511. -- Resolution: Invalid > Spark Streaming Driver Memory > - > >

[jira] [Assigned] (SPARK-27514) Empty window expression results in error in optimizer

2019-04-19 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27514: --- Assignee: Yifei Huang > Empty window expression results in error in optimizer >

[jira] [Resolved] (SPARK-27514) Empty window expression results in error in optimizer

2019-04-19 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27514. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24411

[jira] [Comment Edited] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Praveen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821639#comment-16821639 ] Praveen edited comment on SPARK-27465 at 4/19/19 9:39 AM: -- Hi Shahid, Can you

[jira] [Comment Edited] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821824#comment-16821824 ] Liang-Chi Hsieh edited comment on SPARK-27439 at 4/19/19 10:28 AM: ---

[jira] [Assigned] (SPARK-27504) File source V2: support refreshing metadata cache

2019-04-19 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27504: --- Assignee: Gengliang Wang > File source V2: support refreshing metadata cache >

[jira] [Updated] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27512: - Fix Version/s: (was: 3.0.0) > Decimal parsing leads to unexpected type inference >

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821846#comment-16821846 ] Hyukjin Kwon commented on SPARK-27512: -- You can specify `locale` option from Spark 3.0. > Decimal

[jira] [Resolved] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27512. -- Resolution: Not A Problem > Decimal parsing leads to unexpected type inference >

[jira] [Comment Edited] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Praveen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821639#comment-16821639 ] Praveen edited comment on SPARK-27465 at 4/19/19 10:16 AM: --- Hi Shahid, Can

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Jeff gold (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821840#comment-16821840 ] Jeff gold commented on SPARK-27519: --- Well, unfortunately I don't have access to a higher version of

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821839#comment-16821839 ] Hyukjin Kwon commented on SPARK-27519: -- Can you post a self-contained reproducer and check if the

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2019-04-19 Thread Florian Wilhelm (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821849#comment-16821849 ] Florian Wilhelm commented on SPARK-21187: - I know that this actually does not help with

[jira] [Created] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-04-19 Thread Xingbo Jiang (JIRA)
Xingbo Jiang created SPARK-27520: Summary: Introduce a global config system to replace hadoopConfiguration Key: SPARK-27520 URL: https://issues.apache.org/jira/browse/SPARK-27520 Project: Spark

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-04-19 Thread Xingbo Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821805#comment-16821805 ] Xingbo Jiang commented on SPARK-27520: -- cc [~Ngone51] Would you like to pick up this? > Introduce

[jira] [Resolved] (SPARK-27507) get_json_object fails somewhat arbitrarily on long input

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27507. -- Resolution: Cannot Reproduce {code} Input length: 2264 Output length: 2264 Input length:

[jira] [Comment Edited] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821653#comment-16821653 ] Liang-Chi Hsieh edited comment on SPARK-27367 at 4/19/19 8:01 AM: -- I do

[jira] [Created] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Jeff gold (JIRA)
Jeff gold created SPARK-27519: - Summary: Pandas udf corrupting data Key: SPARK-27519 URL: https://issues.apache.org/jira/browse/SPARK-27519 Project: Spark Issue Type: Bug Components:

[jira] [Updated] (SPARK-27505) autoBroadcastJoinThreshold including bigger table

2019-04-19 Thread Mike Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Chan updated SPARK-27505: -- Issue Type: Bug (was: Question) > autoBroadcastJoinThreshold including bigger table >

[jira] [Commented] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821824#comment-16821824 ] Liang-Chi Hsieh commented on SPARK-27439: - The review is resolved during analysis stage when we

[jira] [Comment Edited] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821824#comment-16821824 ] Liang-Chi Hsieh edited comment on SPARK-27439 at 4/19/19 10:36 AM: ---

[jira] [Comment Edited] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821824#comment-16821824 ] Liang-Chi Hsieh edited comment on SPARK-27439 at 4/19/19 10:35 AM: ---

[jira] [Commented] (SPARK-27509) enable connection in cluster mode for output in client machine

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821848#comment-16821848 ] Hyukjin Kwon commented on SPARK-27509: -- Let's ask it to mailing list first to develop a concrete

[jira] [Resolved] (SPARK-27509) enable connection in cluster mode for output in client machine

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27509. -- Resolution: Incomplete > enable connection in cluster mode for output in client machine >

[jira] [Updated] (SPARK-27509) enable connection in cluster mode for output in client machine

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27509: - Target Version/s: (was: 3.0.0) > enable connection in cluster mode for output in client

[jira] [Commented] (SPARK-27487) Spark - Scala 2.12 compatibility

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821863#comment-16821863 ] Hyukjin Kwon commented on SPARK-27487: -- I think you should ask to Scala side if they have

[jira] [Resolved] (SPARK-27461) Not throwing error for Datatype mismatch

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27461. -- Resolution: Invalid Please ask questions into mailing list. > Not throwing error for

[jira] [Updated] (SPARK-27461) Not throwing error for Datatype mismatch

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27461: - Priority: Major (was: Critical) > Not throwing error for Datatype mismatch >

[jira] [Resolved] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27439. -- Resolution: Not A Problem I agree. > createOrReplaceTempView cannot update old dataset >

[jira] [Reopened] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-27439: -- > createOrReplaceTempView cannot update old dataset >

[jira] [Commented] (SPARK-27442) ParquetFileFormat fails to read column named with invalid characters

2019-04-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-27442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821901#comment-16821901 ] Jan Vršovský commented on SPARK-27442: -- [~hyukjin.kwon] Sorry, I forgot about this... You are

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark, 

[jira] [Resolved] (SPARK-27478) Make HasParallelism public?

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27478. -- Resolution: Invalid Please ask questions to mailing list first before filing an issue. >

[jira] [Resolved] (SPARK-27442) ParquetFileFormat fails to read column named with invalid characters

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27442. -- Resolution: Won't Fix > ParquetFileFormat fails to read column named with invalid characters

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-04-19 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821916#comment-16821916 ] wuyi commented on SPARK-27520: -- [~jiangxb1987] okay, let me try it. thanks. > Introduce a global config

[jira] [Created] (SPARK-27521) move data source v2 API to catalyst module

2019-04-19 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27521: --- Summary: move data source v2 API to catalyst module Key: SPARK-27521 URL: https://issues.apache.org/jira/browse/SPARK-27521 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0

2019-04-19 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822045#comment-16822045 ] Imran Rashid commented on SPARK-27367: -- Did you change spark code as well, to use the new suggested

[jira] [Resolved] (SPARK-27504) File source V2: support refreshing metadata cache

2019-04-19 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27504. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24401

[jira] [Resolved] (SPARK-27487) Spark - Scala 2.12 compatibility

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27487. -- Resolution: Invalid > Spark - Scala 2.12 compatibility > > >

[jira] [Commented] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821868#comment-16821868 ] Hyukjin Kwon commented on SPARK-27471: -- (Fix version is usually set when it's actually fixed, and

[jira] [Updated] (SPARK-27442) ParquetFileFormat fails to read column named with invalid characters

2019-04-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-27442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Vršovský updated SPARK-27442: - Priority: Minor (was: Major) > ParquetFileFormat fails to read column named with invalid

[jira] [Assigned] (SPARK-27486) Enable History server storage information test

2019-04-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27486: - Assignee: shahid > Enable History server storage information test >

[jira] [Resolved] (SPARK-27486) Enable History server storage information test

2019-04-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27486. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24390

[jira] [Commented] (SPARK-27505) autoBroadcastJoinThreshold including bigger table

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821855#comment-16821855 ] Hyukjin Kwon commented on SPARK-27505: -- Can you make a self-reproducer please? Otherwise, virtually

[jira] [Commented] (SPARK-27492) High level user documentation

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821857#comment-16821857 ] Hyukjin Kwon commented on SPARK-27492: -- I was wondering what "this feature" means for a while :) ..

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821859#comment-16821859 ] Hyukjin Kwon commented on SPARK-27491: -- [~toopt4] don't set a blocker which is usually reserved for

[jira] [Updated] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27491: - Priority: Major (was: Blocker) > SPARK REST API - "org.apache.spark.deploy.SparkSubmit

[jira] [Resolved] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27465. -- Resolution: Invalid > Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package >

[jira] [Commented] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821873#comment-16821873 ] Hyukjin Kwon commented on SPARK-27465: -- {{KafkaTestUtils}} is in test sources. It's not meant to be

[jira] [Updated] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2019-04-19 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27497: -- Description: The bucket spec gets wiped out after Spark writes to a Hive-bucketed table that

[jira] [Updated] (SPARK-27498) Built-in parquet code path (convertMetastoreParquet=true) does not respect hive.enforce.bucketing

2019-04-19 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27498: -- Summary: Built-in parquet code path (convertMetastoreParquet=true) does not respect

[jira] [Updated] (SPARK-27466) LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27466: - Description: *1. Create a table in Hive:*   {code:java} CREATE TABLE tab1(   col1

[jira] [Updated] (SPARK-27466) LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27466: - Description: *1. Create a table in Hive:*   {code} CREATE TABLE tab1(   col1 varchar(1),  

[jira] [Commented] (SPARK-27485) Certain query plans fail to run when autoBroadcastJoinThreshold is set to -1

2019-04-19 Thread Muthu Jayakumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821869#comment-16821869 ] Muthu Jayakumar commented on SPARK-27485: - Let me try to build a sql expression for this. What I

[jira] [Commented] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821870#comment-16821870 ] Hyukjin Kwon commented on SPARK-27465: -- Please avoid to set Critical+ which is usually reserved for

[jira] [Updated] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27465: - Description: Hi Team, We are getting the below exceptions with Kafka Client Version 0.11.0.0 

[jira] [Updated] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27465: - Priority: Major (was: Critical) > Kafka Client 0.11.0.0 is not Supporting the kafkatestutils

[jira] [Resolved] (SPARK-27432) Spark job stuck when no jobs/stages are pending

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27432. -- Resolution: Cannot Reproduce > Spark job stuck when no jobs/stages are pending >

[jira] [Commented] (SPARK-27485) Certain query plans fail to run when autoBroadcastJoinThreshold is set to -1

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821865#comment-16821865 ] Hyukjin Kwon commented on SPARK-27485: -- Yes, please share the reproducer > Certain query plans

[jira] [Updated] (SPARK-27447) Add collaborate filtering Explain API in SPARKML

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27447: - Affects Version/s: (was: 2.5.0) 3.0.0 > Add collaborate filtering

[jira] [Commented] (SPARK-27492) High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821900#comment-16821900 ] Thomas Graves commented on SPARK-27492: --- Sorry, it is under the epic and didn't realize it didn't

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Summary: GPU scheduling - High level user documentation (was: High level user documentation)

[jira] [Commented] (SPARK-27487) Spark - Scala 2.12 compatibility

2019-04-19 Thread Vadym Holubnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821911#comment-16821911 ] Vadym Holubnychyi commented on SPARK-27487: --- It's said that 2.12.8 is compatible with all 2.12

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821954#comment-16821954 ] koert kuipers commented on SPARK-27512: --- {code:bash} $ hadoop fs -cat test.bsv x|y 1|1,2,3 2|4,5,6

[jira] [Commented] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821977#comment-16821977 ] Liang-Chi Hsieh commented on SPARK-27439: - One possible issue I'm aware of is, {{df.explain}}

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Jeff gold (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821979#comment-16821979 ] Jeff gold commented on SPARK-27519: --- Ok, i will write a reproducer in python 2 and test it in my

[jira] [Created] (SPARK-27522) Test migration from INT96 to TIMESTAMP_MICROS in parquet

2019-04-19 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27522: -- Summary: Test migration from INT96 to TIMESTAMP_MICROS in parquet Key: SPARK-27522 URL: https://issues.apache.org/jira/browse/SPARK-27522 Project: Spark Issue

[jira] [Updated] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27471: - Fix Version/s: (was: 3.0.0) > Reorganize public v2 catalog API >

[jira] [Resolved] (SPARK-27429) [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27429. -- Resolution: Won't Fix > [SQL] to_timestamp function with additional argument flag that will

[jira] [Assigned] (SPARK-27276) Increase the minimum pyarrow version to 0.12.1

2019-04-19 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp reassigned SPARK-27276: --- Assignee: shane knapp > Increase the minimum pyarrow version to 0.12.1 >

[jira] [Resolved] (SPARK-27276) Increase the minimum pyarrow version to 0.12.1

2019-04-19 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-27276. - Resolution: Fixed > Increase the minimum pyarrow version to 0.12.1 >

[jira] [Created] (SPARK-27523) Resolve scheme-less event log directory relative to default filesystem

2019-04-19 Thread Mikayla Konst (JIRA)
Mikayla Konst created SPARK-27523: - Summary: Resolve scheme-less event log directory relative to default filesystem Key: SPARK-27523 URL: https://issues.apache.org/jira/browse/SPARK-27523 Project:

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:28 PM:

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves commented on SPARK-27495: --- Unfortunately the link to the original design doc was

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:29 PM:

[jira] [Resolved] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

2019-04-19 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-25079. - Resolution: Fixed this is finally done for all branches! > [PYTHON] upgrade python 3.4 -> 3.6

[jira] [Updated] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-27471: -- Target Version/s: 3.0.0 > Reorganize public v2 catalog API > > >

[jira] [Commented] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822271#comment-16822271 ] Ryan Blue commented on SPARK-27471: --- Thanks [~hyukjin.kwon]. I meant to set the target version, not

[jira] [Updated] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27524: Description: The Parquet file format is the default data source to use in input/output. we should

[jira] [Updated] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27524: Description: The Parquet file format is the default data source to use in input/output. we should

[jira] [Updated] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27524: Description: The Parquet file format is the default data source to use in input/output. we should

[jira] [Created] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27524: --- Summary: Remove the parquet-provided support Key: SPARK-27524 URL: https://issues.apache.org/jira/browse/SPARK-27524 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822367#comment-16822367 ] Xiangrui Meng commented on SPARK-27396: --- [~revans2] What would end users do with public APIs for