[jira] [Created] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Jeff gold (JIRA)
Jeff gold created SPARK-27519: - Summary: Pandas udf corrupting data Key: SPARK-27519 URL: https://issues.apache.org/jira/browse/SPARK-27519 Project: Spark Issue Type: Bug Components: Py

[jira] [Comment Edited] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821653#comment-16821653 ] Liang-Chi Hsieh edited comment on SPARK-27367 at 4/19/19 8:01 AM:

[jira] [Commented] (SPARK-27429) [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821763#comment-16821763 ] Liang-Chi Hsieh commented on SPARK-27429: - Generally I think you can always know

[jira] [Updated] (SPARK-27505) autoBroadcastJoinThreshold including bigger table

2019-04-19 Thread Mike Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Chan updated SPARK-27505: -- Issue Type: Bug (was: Question) > autoBroadcastJoinThreshold including bigger table > ---

[jira] [Created] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-04-19 Thread Xingbo Jiang (JIRA)
Xingbo Jiang created SPARK-27520: Summary: Introduce a global config system to replace hadoopConfiguration Key: SPARK-27520 URL: https://issues.apache.org/jira/browse/SPARK-27520 Project: Spark

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-04-19 Thread Xingbo Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821805#comment-16821805 ] Xingbo Jiang commented on SPARK-27520: -- cc [~Ngone51] Would you like to pick up thi

[jira] [Comment Edited] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Praveen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821639#comment-16821639 ] Praveen edited comment on SPARK-27465 at 4/19/19 9:39 AM: -- Hi S

[jira] [Commented] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821824#comment-16821824 ] Liang-Chi Hsieh commented on SPARK-27439: - The review is resolved during analysi

[jira] [Comment Edited] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Praveen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821639#comment-16821639 ] Praveen edited comment on SPARK-27465 at 4/19/19 10:16 AM: --- Hi

[jira] [Assigned] (SPARK-27504) File source V2: support refreshing metadata cache

2019-04-19 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27504: --- Assignee: Gengliang Wang > File source V2: support refreshing metadata cache >

[jira] [Comment Edited] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821824#comment-16821824 ] Liang-Chi Hsieh edited comment on SPARK-27439 at 4/19/19 10:28 AM: ---

[jira] [Comment Edited] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821824#comment-16821824 ] Liang-Chi Hsieh edited comment on SPARK-27439 at 4/19/19 10:35 AM: ---

[jira] [Comment Edited] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821824#comment-16821824 ] Liang-Chi Hsieh edited comment on SPARK-27439 at 4/19/19 10:36 AM: ---

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821839#comment-16821839 ] Hyukjin Kwon commented on SPARK-27519: -- Can you post a self-contained reproducer an

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Jeff gold (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821840#comment-16821840 ] Jeff gold commented on SPARK-27519: --- Well, unfortunately I don't have access to a high

[jira] [Commented] (SPARK-27517) python.PythonRDD: Error while sending iterator

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821843#comment-16821843 ] Hyukjin Kwon commented on SPARK-27517: -- Please just don't copy and paste the error

[jira] [Resolved] (SPARK-27517) python.PythonRDD: Error while sending iterator

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27517. -- Resolution: Incomplete > python.PythonRDD: Error while sending iterator > ---

[jira] [Resolved] (SPARK-27516) java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27516. -- Resolution: Incomplete > java.util.concurrent.TimeoutException: Futures timed out after [1

[jira] [Commented] (SPARK-27516) java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821844#comment-16821844 ] Hyukjin Kwon commented on SPARK-27516: -- I can't reproduce this. I suspect this envi

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821845#comment-16821845 ] Hyukjin Kwon commented on SPARK-27519: -- I can run if you post it in higher versions

[jira] [Updated] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27512: - Fix Version/s: (was: 3.0.0) > Decimal parsing leads to unexpected type inference > -

[jira] [Resolved] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27512. -- Resolution: Not A Problem > Decimal parsing leads to unexpected type inference > -

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821846#comment-16821846 ] Hyukjin Kwon commented on SPARK-27512: -- You can specify `locale` option from Spark

[jira] [Commented] (SPARK-27511) Spark Streaming Driver Memory

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821847#comment-16821847 ] Hyukjin Kwon commented on SPARK-27511: -- Let's ask questions into mailing lists rath

[jira] [Resolved] (SPARK-27511) Spark Streaming Driver Memory

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27511. -- Resolution: Invalid > Spark Streaming Driver Memory > - > >

[jira] [Updated] (SPARK-27509) enable connection in cluster mode for output in client machine

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27509: - Target Version/s: (was: 3.0.0) > enable connection in cluster mode for output in client machin

[jira] [Commented] (SPARK-27509) enable connection in cluster mode for output in client machine

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821848#comment-16821848 ] Hyukjin Kwon commented on SPARK-27509: -- Let's ask it to mailing list first to devel

[jira] [Resolved] (SPARK-27509) enable connection in cluster mode for output in client machine

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27509. -- Resolution: Incomplete > enable connection in cluster mode for output in client machine >

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2019-04-19 Thread Florian Wilhelm (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821849#comment-16821849 ] Florian Wilhelm commented on SPARK-21187: - I know that this actually does not he

[jira] [Resolved] (SPARK-27507) get_json_object fails somewhat arbitrarily on long input

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27507. -- Resolution: Cannot Reproduce {code} Input length: 2264 Output length: 2264 Input length:

[jira] [Commented] (SPARK-27505) autoBroadcastJoinThreshold including bigger table

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821855#comment-16821855 ] Hyukjin Kwon commented on SPARK-27505: -- Can you make a self-reproducer please? Othe

[jira] [Commented] (SPARK-27492) High level user documentation

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821857#comment-16821857 ] Hyukjin Kwon commented on SPARK-27492: -- I was wondering what "this feature" means f

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821859#comment-16821859 ] Hyukjin Kwon commented on SPARK-27491: -- [~toopt4] don't set a blocker which is usua

[jira] [Updated] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27491: - Priority: Major (was: Blocker) > SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status

[jira] [Commented] (SPARK-27487) Spark - Scala 2.12 compatibility

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821863#comment-16821863 ] Hyukjin Kwon commented on SPARK-27487: -- I think you should ask to Scala side if the

[jira] [Resolved] (SPARK-27504) File source V2: support refreshing metadata cache

2019-04-19 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27504. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24401 [https://gith

[jira] [Resolved] (SPARK-27487) Spark - Scala 2.12 compatibility

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27487. -- Resolution: Invalid > Spark - Scala 2.12 compatibility > > >

[jira] [Commented] (SPARK-27485) Certain query plans fail to run when autoBroadcastJoinThreshold is set to -1

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821865#comment-16821865 ] Hyukjin Kwon commented on SPARK-27485: -- Yes, please share the reproducer > Certain

[jira] [Resolved] (SPARK-27478) Make HasParallelism public?

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27478. -- Resolution: Invalid Please ask questions to mailing list first before filing an issue. > Make

[jira] [Updated] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27471: - Fix Version/s: (was: 3.0.0) > Reorganize public v2 catalog API > ---

[jira] [Commented] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821868#comment-16821868 ] Hyukjin Kwon commented on SPARK-27471: -- (Fix version is usually set when it's actua

[jira] [Updated] (SPARK-27466) LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27466: - Description: *1. Create a table in Hive:*   {code} CREATE TABLE tab1(   col1 varchar(1),  

[jira] [Commented] (SPARK-27485) Certain query plans fail to run when autoBroadcastJoinThreshold is set to -1

2019-04-19 Thread Muthu Jayakumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821869#comment-16821869 ] Muthu Jayakumar commented on SPARK-27485: - Let me try to build a sql expression

[jira] [Updated] (SPARK-27466) LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27466: - Description: *1. Create a table in Hive:*   {code:java} CREATE TABLE tab1(   col1 varchar(1),

[jira] [Commented] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821870#comment-16821870 ] Hyukjin Kwon commented on SPARK-27465: -- Please avoid to set Critical+ which is usua

[jira] [Updated] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27465: - Description: Hi Team, We are getting the below exceptions with Kafka Client Version 0.11.0.0 fo

[jira] [Updated] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27465: - Priority: Major (was: Critical) > Kafka Client 0.11.0.0 is not Supporting the kafkatestutils pa

[jira] [Commented] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821873#comment-16821873 ] Hyukjin Kwon commented on SPARK-27465: -- {{KafkaTestUtils}} is in test sources. It's

[jira] [Resolved] (SPARK-27465) Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27465. -- Resolution: Invalid > Kafka Client 0.11.0.0 is not Supporting the kafkatestutils package > ---

[jira] [Resolved] (SPARK-27461) Not throwing error for Datatype mismatch

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27461. -- Resolution: Invalid Please ask questions into mailing list. > Not throwing error for Datatype

[jira] [Updated] (SPARK-27461) Not throwing error for Datatype mismatch

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27461: - Priority: Major (was: Critical) > Not throwing error for Datatype mismatch > --

[jira] [Updated] (SPARK-27447) Add collaborate filtering Explain API in SPARKML

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27447: - Affects Version/s: (was: 2.5.0) 3.0.0 > Add collaborate filtering Exp

[jira] [Resolved] (SPARK-27442) ParquetFileFormat fails to read column named with invalid characters

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27442. -- Resolution: Won't Fix > ParquetFileFormat fails to read column named with invalid characters >

[jira] [Resolved] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27439. -- Resolution: Not A Problem I agree. > createOrReplaceTempView cannot update old dataset >

[jira] [Reopened] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-27439: -- > createOrReplaceTempView cannot update old dataset >

[jira] [Resolved] (SPARK-27432) Spark job stuck when no jobs/stages are pending

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27432. -- Resolution: Cannot Reproduce > Spark job stuck when no jobs/stages are pending > -

[jira] [Resolved] (SPARK-27429) [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast

2019-04-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27429. -- Resolution: Won't Fix > [SQL] to_timestamp function with additional argument flag that will al

[jira] [Commented] (SPARK-27492) High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821900#comment-16821900 ] Thomas Graves commented on SPARK-27492: --- Sorry, it is under the epic and didn't re

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Summary: GPU scheduling - High level user documentation (was: High level user documentation)

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark,  https://issues.apache

[jira] [Commented] (SPARK-27442) ParquetFileFormat fails to read column named with invalid characters

2019-04-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-27442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821901#comment-16821901 ] Jan Vršovský commented on SPARK-27442: -- [~hyukjin.kwon] Sorry, I forgot about this.

[jira] [Updated] (SPARK-27442) ParquetFileFormat fails to read column named with invalid characters

2019-04-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-27442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Vršovský updated SPARK-27442: - Priority: Minor (was: Major) > ParquetFileFormat fails to read column named with invalid charac

[jira] [Commented] (SPARK-27487) Spark - Scala 2.12 compatibility

2019-04-19 Thread Vadym Holubnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821911#comment-16821911 ] Vadym Holubnychyi commented on SPARK-27487: --- It's said that 2.12.8 is compatib

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-04-19 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821916#comment-16821916 ] wuyi commented on SPARK-27520: -- [~jiangxb1987] okay, let me try it. thanks. > Introduce a

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

2019-04-19 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821954#comment-16821954 ] koert kuipers commented on SPARK-27512: --- {code:bash} $ hadoop fs -cat test.bsv x|y

[jira] [Updated] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2019-04-19 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27497: -- Description: The bucket spec gets wiped out after Spark writes to a Hive-bucketed table that

[jira] [Commented] (SPARK-27519) Pandas udf corrupting data

2019-04-19 Thread Jeff gold (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821979#comment-16821979 ] Jeff gold commented on SPARK-27519: --- Ok, i will write a reproducer in python 2 and tes

[jira] [Commented] (SPARK-27439) createOrReplaceTempView cannot update old dataset

2019-04-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821977#comment-16821977 ] Liang-Chi Hsieh commented on SPARK-27439: - One possible issue I'm aware of is, {

[jira] [Created] (SPARK-27521) move data source v2 API to catalyst module

2019-04-19 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27521: --- Summary: move data source v2 API to catalyst module Key: SPARK-27521 URL: https://issues.apache.org/jira/browse/SPARK-27521 Project: Spark Issue Type: Improvem

[jira] [Assigned] (SPARK-27486) Enable History server storage information test

2019-04-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27486: - Assignee: shahid > Enable History server storage information test > ---

[jira] [Resolved] (SPARK-27486) Enable History server storage information test

2019-04-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27486. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24390 [https://github.c

[jira] [Updated] (SPARK-27498) Built-in parquet code path (convertMetastoreParquet=true) does not respect hive.enforce.bucketing

2019-04-19 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-27498: -- Summary: Built-in parquet code path (convertMetastoreParquet=true) does not respect hive.enfor

[jira] [Created] (SPARK-27522) Test migration from INT96 to TIMESTAMP_MICROS in parquet

2019-04-19 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27522: -- Summary: Test migration from INT96 to TIMESTAMP_MICROS in parquet Key: SPARK-27522 URL: https://issues.apache.org/jira/browse/SPARK-27522 Project: Spark Issue Ty

[jira] [Commented] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0

2019-04-19 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822045#comment-16822045 ] Imran Rashid commented on SPARK-27367: -- Did you change spark code as well, to use t

[jira] [Resolved] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

2019-04-19 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-25079. - Resolution: Fixed this is finally done for all branches! > [PYTHON] upgrade python 3.4 -> 3.6 >

[jira] [Assigned] (SPARK-27276) Increase the minimum pyarrow version to 0.12.1

2019-04-19 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp reassigned SPARK-27276: --- Assignee: shane knapp > Increase the minimum pyarrow version to 0.12.1 > --

[jira] [Resolved] (SPARK-27276) Increase the minimum pyarrow version to 0.12.1

2019-04-19 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-27276. - Resolution: Fixed > Increase the minimum pyarrow version to 0.12.1 > ---

[jira] [Created] (SPARK-27523) Resolve scheme-less event log directory relative to default filesystem

2019-04-19 Thread Mikayla Konst (JIRA)
Mikayla Konst created SPARK-27523: - Summary: Resolve scheme-less event log directory relative to default filesystem Key: SPARK-27523 URL: https://issues.apache.org/jira/browse/SPARK-27523 Project: Spa

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves commented on SPARK-27495: --- Unfortunately the link to the original de

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:28 PM: --

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:29 PM: --

[jira] [Updated] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-27471: -- Target Version/s: 3.0.0 > Reorganize public v2 catalog API > > >

[jira] [Commented] (SPARK-27471) Reorganize public v2 catalog API

2019-04-19 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822271#comment-16822271 ] Ryan Blue commented on SPARK-27471: --- Thanks [~hyukjin.kwon]. I meant to set the target

[jira] [Created] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27524: --- Summary: Remove the parquet-provided support Key: SPARK-27524 URL: https://issues.apache.org/jira/browse/SPARK-27524 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27524: Description: The Parquet file format is the default data source to use in input/output. we should

[jira] [Updated] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27524: Description: The Parquet file format is the default data source to use in input/output. we should

[jira] [Updated] (SPARK-27524) Remove the parquet-provided support

2019-04-19 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27524: Description: The Parquet file format is the default data source to use in input/output. we should

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822367#comment-16822367 ] Xiangrui Meng commented on SPARK-27396: --- [~revans2] What would end users do with p