[jira] [Created] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-04-23 Thread Jepson (JIRA)
Jepson created SPARK-27554: -- Summary: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] Key: SPARK-27554 URL: https://issues.apache.org/jira/browse/SPARK-27554

[jira] [Created] (SPARK-27553) Operation log is not closed when close session

2019-04-23 Thread pin_zhang (JIRA)
pin_zhang created SPARK-27553: - Summary: Operation log is not closed when close session Key: SPARK-27553 URL: https://issues.apache.org/jira/browse/SPARK-27553 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-27552) The configuration `hive.exec.stagingdir` is invalid on Windows OS

2019-04-23 Thread liuxian (JIRA)
liuxian created SPARK-27552: --- Summary: The configuration `hive.exec.stagingdir` is invalid on Windows OS Key: SPARK-27552 URL: https://issues.apache.org/jira/browse/SPARK-27552 Project: Spark

[jira] [Updated] (SPARK-27551) Uniformative error message for mismatched types in when().otherwise()

2019-04-23 Thread Huon Wilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huon Wilson updated SPARK-27551: Description: When a {{when(...).otherwise(...)}} construct has a type error, the error message

[jira] [Created] (SPARK-27551) Uniformative error message for mismatched types in when().otherwise()

2019-04-23 Thread Huon Wilson (JIRA)
Huon Wilson created SPARK-27551: --- Summary: Uniformative error message for mismatched types in when().otherwise() Key: SPARK-27551 URL: https://issues.apache.org/jira/browse/SPARK-27551 Project: Spark

[jira] [Comment Edited] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-23 Thread Dharmesh Kakadia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824604#comment-16824604 ] Dharmesh Kakadia edited comment on SPARK-27287 at 4/23/19 10:27 PM:

[jira] [Commented] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-23 Thread Dharmesh Kakadia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824604#comment-16824604 ] Dharmesh Kakadia commented on SPARK-27287: -- Understood. I do not know how to "use sparkSession"

[jira] [Updated] (SPARK-27550) Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27550: -- Description: Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test

[jira] [Created] (SPARK-27550) Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

2019-04-23 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-27550: - Summary: Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12 Key: SPARK-27550 URL: https://issues.apache.org/jira/browse/SPARK-27550 Project:

[jira] [Updated] (SPARK-27550) Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27550: -- Issue Type: Bug (was: Task) > Fix `test-dependencies.sh` not to use `kafka-0-8` profile for

[jira] [Assigned] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26970: - Assignee: Andrew Crosby > Can't load PipelineModel that was created in Scala with Python due

[jira] [Created] (SPARK-27549) Commit Kafka Source offsets to facilitate external tooling

2019-04-23 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-27549: --- Summary: Commit Kafka Source offsets to facilitate external tooling Key: SPARK-27549 URL: https://issues.apache.org/jira/browse/SPARK-27549 Project:

[jira] [Commented] (SPARK-27534) Do not load `content` column in binary data source if it is not selected

2019-04-23 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824554#comment-16824554 ] Weichen Xu commented on SPARK-27534: I am working on this. :) > Do not load `content` column in

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824550#comment-16824550 ] Dongjoon Hyun commented on SPARK-25954: --- Yep. [~ijuma]. I updated Spark already via SPARK-27260.

[jira] [Resolved] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-04-23 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-26970. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24426

[jira] [Created] (SPARK-27548) PySpark toLocalIterator does not raise errors from worker

2019-04-23 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-27548: Summary: PySpark toLocalIterator does not raise errors from worker Key: SPARK-27548 URL: https://issues.apache.org/jira/browse/SPARK-27548 Project: Spark

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824226#comment-16824226 ] Ismael Juma commented on SPARK-25954: - That's awesome, sorry for the noise. > Upgrade to Kafka

[jira] [Resolved] (SPARK-27128) Migrate JSON to File Data Source V2

2019-04-23 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27128. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24058

[jira] [Assigned] (SPARK-27128) Migrate JSON to File Data Source V2

2019-04-23 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27128: --- Assignee: Gengliang Wang > Migrate JSON to File Data Source V2 >

[jira] [Issue Comment Deleted] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

2019-04-23 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivu Sondur updated SPARK-27409: - Comment: was deleted (was: i am checking this) > Micro-batch support for Kafka Source in Spark

[jira] [Issue Comment Deleted] (SPARK-27421) RuntimeException when querying a view on a partitioned parquet table

2019-04-23 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivu Sondur updated SPARK-27421: - Comment: was deleted (was: i am checking this issue) > RuntimeException when querying a view

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824188#comment-16824188 ] Gabor Somogyi commented on SPARK-25954: --- [~ijuma] On the actual master 2.2.0 kafka-clients used.

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824177#comment-16824177 ] Ismael Juma commented on SPARK-25954: - [~dongjoon] Kafka 2.2.0 has a fix for an issue that has

[jira] [Resolved] (SPARK-27503) JobGenerator thread exit for some fatal errors but application keeps running

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27503. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24400

[jira] [Assigned] (SPARK-27503) JobGenerator thread exit for some fatal errors but application keeps running

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27503: - Assignee: Genmao Yu > JobGenerator thread exit for some fatal errors but application keeps

[jira] [Assigned] (SPARK-27481) Upgrade commons-logging to 1.1.3

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27481: - Assignee: Yuming Wang > Upgrade commons-logging to 1.1.3 > >

[jira] [Resolved] (SPARK-27481) Upgrade commons-logging to 1.1.3

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27481. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24388

[jira] [Updated] (SPARK-27481) Upgrade commons-logging to 1.1.3

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27481: -- Priority: Minor (was: Major) > Upgrade commons-logging to 1.1.3 > >

[jira] [Resolved] (SPARK-27544) Fix Python test script to work on Scala-2.12 build

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27544. --- Resolution: Fixed Fix Version/s: 2.4.3 This is resolved via

[jira] [Created] (SPARK-27547) fix DataFrame self-join problems

2019-04-23 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27547: --- Summary: fix DataFrame self-join problems Key: SPARK-27547 URL: https://issues.apache.org/jira/browse/SPARK-27547 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-23 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823876#comment-16823876 ] Marco Gaido commented on SPARK-27287: - [~dharmesh.kakadia] the point is: if you set a config on the 

[jira] [Commented] (SPARK-27542) SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs when using certain legacy OutputFormats

2019-04-23 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823844#comment-16823844 ] Shivu Sondur commented on SPARK-27542: -- [~joshrosen] Can you provide the exact steps to reproduce

[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823801#comment-16823801 ] Jiatao Tao edited comment on SPARK-27546 at 4/23/19 8:17 AM: - see an example

[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823808#comment-16823808 ] Jiatao Tao commented on SPARK-27546: So in my opinion, we should change

[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823801#comment-16823801 ] Jiatao Tao edited comment on SPARK-27546 at 4/23/19 8:10 AM: - see an example

[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823801#comment-16823801 ] Jiatao Tao commented on SPARK-27546: see an example like this:   {code:java} // config spark

[jira] [Created] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
Jiatao Tao created SPARK-27546: -- Summary: Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone Key: SPARK-27546 URL: https://issues.apache.org/jira/browse/SPARK-27546 Project: Spark

[jira] [Updated] (SPARK-27545) Uncache table needs to delete the temporary view created when the cache table is executed.

2019-04-23 Thread hantiantian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hantiantian updated SPARK-27545: Description: spark-sql> cache table v1 as select * from a; spark-sql> uncache table v1;

[jira] [Created] (SPARK-27545) Uncache table needs to delete the temporary view created when the cache table is executed.

2019-04-23 Thread hantiantian (JIRA)
hantiantian created SPARK-27545: --- Summary: Uncache table needs to delete the temporary view created when the cache table is executed. Key: SPARK-27545 URL: https://issues.apache.org/jira/browse/SPARK-27545

[jira] [Assigned] (SPARK-27544) Fix Python test script to work on Scala-2.12 build

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27544: - Assignee: Dongjoon Hyun > Fix Python test script to work on Scala-2.12 build >

[jira] [Updated] (SPARK-27544) Fix Python test script to work on Scala-2.12 build

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27544: -- Description: Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test

[jira] [Created] (SPARK-27544) Fix Python test script to work on Scala-2.12 build

2019-04-23 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-27544: - Summary: Fix Python test script to work on Scala-2.12 build Key: SPARK-27544 URL: https://issues.apache.org/jira/browse/SPARK-27544 Project: Spark Issue