[jira] [Commented] (SPARK-29267) rdd.countApprox should stop when 'timeout'

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949163#comment-16949163 ] Hyukjin Kwon commented on SPARK-29267: -- Where do you use {{countApprox(timeout: Long, confidence:

[jira] [Updated] (SPARK-29422) test failed: org.apache.spark.deploy.history.HistoryServerSuite ajax rendered relative links are prefixed with uiRoot

2019-10-11 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtianhua updated SPARK-29422: - Summary: test failed: org.apache.spark.deploy.history.HistoryServerSuite ajax rendered relative

[jira] [Commented] (SPARK-29241) Show alternative names in the function description

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949171#comment-16949171 ] Hyukjin Kwon commented on SPARK-29241: -- +1 too > Show alternative names in the function

[jira] [Resolved] (SPARK-28653) Create table using DDL statement should not auto create the destination folder

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28653. -- Resolution: Cannot Reproduce > Create table using DDL statement should not auto create the

[jira] [Resolved] (SPARK-28943) NoSuchMethodError: shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28943. -- Resolution: Incomplete > NoSuchMethodError: >

[jira] [Commented] (SPARK-28943) NoSuchMethodError: shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949185#comment-16949185 ] Hyukjin Kwon commented on SPARK-28943: -- Let's leave this resolved for now. 2.2.0 is EOL release and

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread Shimin Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949190#comment-16949190 ] Shimin Yang commented on SPARK-29302: - Hi all, I have also encountered this issue and have a fix

[jira] [Comment Edited] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread Shimin Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949190#comment-16949190 ] Shimin Yang edited comment on SPARK-29302 at 10/11/19 7:16 AM: --- Hi all, I

[jira] [Commented] (SPARK-24540) Support for multiple character delimiter in Spark CSV read

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949161#comment-16949161 ] Hyukjin Kwon commented on SPARK-24540: -- [~maxgekk] I fixed the Jira description a bit. Let's traget

[jira] [Updated] (SPARK-24540) Support for multiple character delimiter in Spark CSV read

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24540: - Description: Currently, the delimiter option Spark 2.0 to read and split CSV files/data only

[jira] [Commented] (SPARK-29340) Spark Sql executions do not use thread local jobgroup

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949165#comment-16949165 ] Hyukjin Kwon commented on SPARK-29340: -- Thanks for your response, [~navdeepniku]. > Spark Sql

[jira] [Resolved] (SPARK-29078) Spark shell fails if read permission is not granted to hive warehouse directory

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29078. -- Resolution: Not A Problem > Spark shell fails if read permission is not granted to hive

[jira] [Commented] (SPARK-29041) Allow createDataFrame to accept bytes as binary type

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949169#comment-16949169 ] Hyukjin Kwon commented on SPARK-29041: -- It was discussed to not backport. See the discussion in the

[jira] [Commented] (SPARK-29047) use spark-submit Not a file: hdfs://

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949182#comment-16949182 ] Hyukjin Kwon commented on SPARK-29047: -- Thanks for your response, [~我心凉凉的]. > use spark-submit

[jira] [Commented] (SPARK-28864) Add spark connector for Alibaba Log Service

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949184#comment-16949184 ] Hyukjin Kwon commented on SPARK-28864: -- I don't think this should be in Spark for now. Let's do it

[jira] [Resolved] (SPARK-28864) Add spark connector for Alibaba Log Service

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28864. -- Resolution: Won't Fix > Add spark connector for Alibaba Log Service >

[jira] [Commented] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-11 Thread pin_zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949187#comment-16949187 ] pin_zhang commented on SPARK-29423: --- The same result on Spark 2.4.3. > leak on

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread Shimin Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949194#comment-16949194 ] Shimin Yang commented on SPARK-29302: - This bug can be easily reproduced by inserting into a

[jira] [Comment Edited] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread Shimin Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949194#comment-16949194 ] Shimin Yang edited comment on SPARK-29302 at 10/11/19 6:56 AM: --- This bug

[jira] [Updated] (SPARK-29382) Support writing `INTERVAL` type to datasource table

2019-10-11 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-29382: --- Summary: Support writing `INTERVAL` type to datasource table (was: Support the `INTERVAL` type by

[jira] [Commented] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-11 Thread antonkulaga (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949230#comment-16949230 ] antonkulaga commented on SPARK-28547: - >I bet there is room for improvement, but, ten thousand

[jira] [Commented] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset

2019-10-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949282#comment-16949282 ] L. C. Hsieh commented on SPARK-29427: - This seems to be an interesting idea. I think currently if

[jira] [Updated] (SPARK-29382) Support writing `INTERVAL` type to datasource table

2019-10-11 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-29382: --- Description: Creating a table with `INTERVAL` column for writing failed with the error: {code:java}

[jira] [Comment Edited] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949229#comment-16949229 ] feiwang edited comment on SPARK-29302 at 10/11/19 7:46 AM: --- [~dangdangdang]

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949229#comment-16949229 ] feiwang commented on SPARK-29302: - [~dangdangdang] Hi, I have thought a simple solution. We just need

[jira] [Closed] (SPARK-6363) Switch to Scala 2.11 for default build

2019-10-11 Thread antonkulaga (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antonkulaga closed SPARK-6363. -- Resolved > Switch to Scala 2.11 for default build > -- > >

[jira] [Updated] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset

2019-10-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-29427: Component/s: (was: Spark Core) > Create KeyValueGroupedDataset from RelationalGroupedDataset

[jira] [Created] (SPARK-29436) Support executor for selecting scheduler through scheduler name

2019-10-11 Thread merrily01 (Jira)
merrily01 created SPARK-29436: - Summary: Support executor for selecting scheduler through scheduler name Key: SPARK-29436 URL: https://issues.apache.org/jira/browse/SPARK-29436 Project: Spark

[jira] [Updated] (SPARK-29436) Support executor for selecting scheduler through scheduler name

2019-10-11 Thread merrily01 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] merrily01 updated SPARK-29436: -- Description: In the case of k8s multi-scheduler, support executor for selecting scheduler through

[jira] [Updated] (SPARK-29436) Support executor for selecting scheduler through scheduler name in the case of k8s multi-scheduler scenario.

2019-10-11 Thread merrily01 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] merrily01 updated SPARK-29436: -- Summary: Support executor for selecting scheduler through scheduler name in the case of k8s

[jira] [Created] (SPARK-29437) CSV Writer should escape 'escapechar' when it exists in the data

2019-10-11 Thread Tomasz Bartczak (Jira)
Tomasz Bartczak created SPARK-29437: --- Summary: CSV Writer should escape 'escapechar' when it exists in the data Key: SPARK-29437 URL: https://issues.apache.org/jira/browse/SPARK-29437 Project:

[jira] [Commented] (SPARK-29428) Can't persist/set None-valued param

2019-10-11 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949586#comment-16949586 ] Huaxin Gao commented on SPARK-29428: I will take a look of this. > Can't persist/set None-valued

[jira] [Updated] (SPARK-29442) Set `default` mode should override the existing mode

2019-10-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29442: -- Summary: Set `default` mode should override the existing mode (was: Last `mode("default")`

[jira] [Created] (SPARK-29442) Last `mode("default")` should work like `mode("error")`

2019-10-11 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29442: - Summary: Last `mode("default")` should work like `mode("error")` Key: SPARK-29442 URL: https://issues.apache.org/jira/browse/SPARK-29442 Project: Spark

[jira] [Assigned] (SPARK-25341) Support rolling back a shuffle map stage and re-generate the shuffle files

2019-10-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-25341: --- Assignee: Yuanjian Li > Support rolling back a shuffle map stage and re-generate the

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-11 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949765#comment-16949765 ] Luca Canali commented on SPARK-29396: - This looks quite a good improvement proposal to me.

[jira] [Commented] (SPARK-28867) InMemoryStore checkpoint to speed up replay log file in HistoryServer

2019-10-11 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949767#comment-16949767 ] Imran Rashid commented on SPARK-28867: -- This is closely related to SPARK-20656. Its not *quite* a

[jira] [Assigned] (SPARK-29148) Modify dynamic allocation manager for stage level scheduling

2019-10-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29148: - Assignee: Thomas Graves > Modify dynamic allocation manager for stage level scheduling

[jira] [Assigned] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

2019-10-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29415: - Assignee: Thomas Graves > Stage Level Sched: Add base ResourceProfile and Request

[jira] [Assigned] (SPARK-29306) Executors need to track what ResourceProfile they are created with

2019-10-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29306: - Assignee: Thomas Graves > Executors need to track what ResourceProfile they are

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-11 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949522#comment-16949522 ] Sandeep Katta commented on SPARK-29435: --- Ya I am even able to reproduce it, I am looking into this

[jira] [Updated] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29423: - Affects Version/s: 2.4.3 > leak on

[jira] [Commented] (SPARK-27981) Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`

2019-10-11 Thread Kuhu Shukla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949598#comment-16949598 ] Kuhu Shukla commented on SPARK-27981: - Hello [~dongjoon], I am seeing the following Warning when

[jira] [Created] (SPARK-29443) Spark-on-Kubernetes: driver pod keeps spinning/terminating executor pods

2019-10-11 Thread James Nguyen (Jira)
James Nguyen created SPARK-29443: Summary: Spark-on-Kubernetes: driver pod keeps spinning/terminating executor pods Key: SPARK-29443 URL: https://issues.apache.org/jira/browse/SPARK-29443 Project:

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949554#comment-16949554 ] Hyukjin Kwon commented on SPARK-16534: -- No, it was resolved as discussed about affected versions

[jira] [Commented] (SPARK-28547) Make it work for wide (> 10K columns data)

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949561#comment-16949561 ] Hyukjin Kwon commented on SPARK-28547: -- [~antonkulaga], if you're unable to elaborate what to

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-11 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949508#comment-16949508 ] Sandeep Katta commented on SPARK-29435: --- [~koert] I will investigate this further, If I found root

[jira] [Created] (SPARK-29441) Unable to Alter table column type in spark.

2019-10-11 Thread prudhviraj (Jira)
prudhviraj created SPARK-29441: -- Summary: Unable to Alter table column type in spark. Key: SPARK-29441 URL: https://issues.apache.org/jira/browse/SPARK-29441 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-11 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Issue Type: Bug (was: Question) > nullable flag of new column

[jira] [Comment Edited] (SPARK-16534) Kafka 0.10 Python support

2019-10-11 Thread Rakesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949521#comment-16949521 ] Rakesh edited comment on SPARK-16534 at 10/11/19 2:59 PM: -- Was this actually

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-11 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949669#comment-16949669 ] Imran Rashid commented on SPARK-29396: -- The memory monitor plugin I wrote (why I pushed the

[jira] [Reopened] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-11 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan reopened SPARK-29432: -- Edited the commands to reproduce the issue. The issue is not

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2019-10-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949556#comment-16949556 ] Hyukjin Kwon commented on SPARK-16534: -- Actually, this JIRA seems not going to be fixed per the

[jira] [Updated] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-11 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Saraswathi Krishnan updated SPARK-29432: - Description: When I add a new column to a dataframe with

[jira] [Issue Comment Deleted] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-11 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-29435: -- Comment: was deleted (was: [~koert] I will investigate this further, If I found root cause

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2019-10-11 Thread Rakesh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949521#comment-16949521 ] Rakesh commented on SPARK-16534: Was this actually resolved? Any when the support will be available?  

[jira] [Updated] (SPARK-29443) Spark-on-Kubernetes: driver pod keeps spinning/terminating executor pods

2019-10-11 Thread James Nguyen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Nguyen updated SPARK-29443: - Environment: EKS version: v1.14.6-eks-5047ed with 18 x r5a.12xlarge EC2 instances as Worker

[jira] [Commented] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-11 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949678#comment-16949678 ] Prasanna Saraswathi Krishnan commented on SPARK-29432: -- Hello I corrected the

[jira] [Comment Edited] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-11 Thread Prasanna Saraswathi Krishnan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949068#comment-16949068 ] Prasanna Saraswathi Krishnan edited comment on SPARK-29432 at 10/11/19 5:51 PM:

[jira] [Updated] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-10-11 Thread Ilan Filonenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilan Filonenko updated SPARK-27812: --- Affects Version/s: 2.4.4 > kubernetes client import non-daemon thread which block jvm exit.

[jira] [Commented] (SPARK-29295) Duplicate result when dropping partition of an external table and then overwriting

2019-10-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949886#comment-16949886 ] angerszhu commented on SPARK-29295: ---  This probelm seems start from hive 1.2 I test in our env

[jira] [Commented] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

2019-10-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949910#comment-16949910 ] L. C. Hsieh commented on SPARK-29432: - You can find related discussion like:

[jira] [Commented] (SPARK-12014) Spark SQL query containing semicolon is broken in Beeline (related to HIVE-11100)

2019-10-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949874#comment-16949874 ] Yuming Wang commented on SPARK-12014: - [~chutium] This issue has been fixed since we upgraded 

[jira] [Commented] (SPARK-29428) Can't persist/set None-valued param

2019-10-11 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949908#comment-16949908 ] L. C. Hsieh commented on SPARK-29428: - Yeah, I think it is not valid to set None to input column

[jira] [Assigned] (SPARK-29402) Add tests for grouped map pandas_udf using window

2019-10-11 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-29402: Assignee: Bryan Cutler > Add tests for grouped map pandas_udf using window >

[jira] [Resolved] (SPARK-29402) Add tests for grouped map pandas_udf using window

2019-10-11 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-29402. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26063

[jira] [Commented] (SPARK-29428) Can't persist/set None-valued param

2019-10-11 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949863#comment-16949863 ] Huaxin Gao commented on SPARK-29428: This seems to be a usage error. The error was issued at this

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-11 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949881#comment-16949881 ] Sandeep Katta commented on SPARK-29435: --- [~koert] can you please apply this

[jira] [Commented] (SPARK-29426) Watermark does not take effect

2019-10-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949892#comment-16949892 ] Jungtaek Lim commented on SPARK-29426: -- Could you attach query listener and see the actual event

[jira] [Created] (SPARK-29444) Add configuration to support JacksonGenrator to keep fields with null values

2019-10-11 Thread Jackey Lee (Jira)
Jackey Lee created SPARK-29444: -- Summary: Add configuration to support JacksonGenrator to keep fields with null values Key: SPARK-29444 URL: https://issues.apache.org/jira/browse/SPARK-29444 Project:

[jira] [Commented] (SPARK-29222) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_parameter_convergence

2019-10-11 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949869#comment-16949869 ] huangtianhua commented on SPARK-29222: -- [~hyukjin.kwon], we took test to increase default timeout

[jira] [Commented] (SPARK-29421) Add an opportunity to change the file format of command CREATE TABLE LIKE

2019-10-11 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949899#comment-16949899 ] Lantao Jin commented on SPARK-29421: PR filed. [~cloud_fan], could you help to review it? > Add an

[jira] [Created] (SPARK-29445) Bump netty-all from 4.1.39.Final to 4.1.42.Final

2019-10-11 Thread Fokko Driesprong (Jira)
Fokko Driesprong created SPARK-29445: Summary: Bump netty-all from 4.1.39.Final to 4.1.42.Final Key: SPARK-29445 URL: https://issues.apache.org/jira/browse/SPARK-29445 Project: Spark

[jira] [Commented] (SPARK-29423) leak on org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus

2019-10-11 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949340#comment-16949340 ] Yuming Wang commented on SPARK-29423: - Maybe we should lazily initialize {{StreamingQueryManager}}.

[jira] [Created] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
Genmao Yu created SPARK-29438: - Summary: Failed to get state store in stream-stream join Key: SPARK-29438 URL: https://issues.apache.org/jira/browse/SPARK-29438 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949365#comment-16949365 ] Genmao Yu commented on SPARK-29438: --- There are several optional alternatives to resolve this issue: *

[jira] [Commented] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949430#comment-16949430 ] Jungtaek Lim commented on SPARK-29438: -- Could you point out the codebase where you are referring?

[jira] [Commented] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949465#comment-16949465 ] Jungtaek Lim commented on SPARK-29438: -- Could you please link the actual code block from Github

[jira] [Updated] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-29438: -- Description: Now, Spark use the `TaskPartitionId` to determine the StateStore path. {code:java}

[jira] [Created] (SPARK-29440) Support java.time.Duration as an external type of CalendarIntervalType

2019-10-11 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-29440: -- Summary: Support java.time.Duration as an external type of CalendarIntervalType Key: SPARK-29440 URL: https://issues.apache.org/jira/browse/SPARK-29440 Project: Spark

[jira] [Commented] (SPARK-29437) CSV Writer should escape 'escapechar' when it exists in the data

2019-10-11 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949483#comment-16949483 ] Ankit Raj Boudh commented on SPARK-29437: - hi @Tomasz Bartczak, I will check this issue. > CSV

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread Shimin Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949359#comment-16949359 ] Shimin Yang commented on SPARK-29302: - Hi [~hzfeiwang], thanks for your contribution.(y) I just

[jira] [Comment Edited] (SPARK-29302) dynamic partition overwrite with speculation enabled

2019-10-11 Thread Shimin Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949190#comment-16949190 ] Shimin Yang edited comment on SPARK-29302 at 10/11/19 10:41 AM: Hi all,

[jira] [Updated] (SPARK-29427) Create KeyValueGroupedDataset in a relational way

2019-10-11 Thread Alexander Hagerf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Hagerf updated SPARK-29427: - Summary: Create KeyValueGroupedDataset in a relational way (was: Create

[jira] [Created] (SPARK-29439) DDL commands should not use DataSourceV2Relation

2019-10-11 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-29439: --- Summary: DDL commands should not use DataSourceV2Relation Key: SPARK-29439 URL: https://issues.apache.org/jira/browse/SPARK-29439 Project: Spark Issue Type:

[jira] [Updated] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated SPARK-29438: -- Description: Now, Spark use the `TaskPartitionId` to determine the StateStore path. {code:java}

[jira] [Comment Edited] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949430#comment-16949430 ] Jungtaek Lim edited comment on SPARK-29438 at 10/11/19 1:12 PM: Could

[jira] [Assigned] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29396: - Assignee: (was: Thomas Graves) > Extend Spark plugin interface to driver >

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949460#comment-16949460 ] Thomas Graves commented on SPARK-29396: ---  While I do like the idea, do we have some concrete use

[jira] [Comment Edited] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949365#comment-16949365 ] Genmao Yu edited comment on SPARK-29438 at 10/11/19 11:15 AM: -- There are

[jira] [Comment Edited] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949365#comment-16949365 ] Genmao Yu edited comment on SPARK-29438 at 10/11/19 11:15 AM: -- There are

[jira] [Updated] (SPARK-29436) Support executor for selecting scheduler through scheduler name in the case of k8s multi-scheduler scenario.

2019-10-11 Thread merrily01 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] merrily01 updated SPARK-29436: -- Issue Type: New Feature (was: Improvement) > Support executor for selecting scheduler through

[jira] [Commented] (SPARK-29438) Failed to get state store in stream-stream join

2019-10-11 Thread Genmao Yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949461#comment-16949461 ] Genmao Yu commented on SPARK-29438: --- [~kabhwan] Yes, the whole statestore path is

[jira] [Assigned] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29396: - Assignee: Thomas Graves > Extend Spark plugin interface to driver >