[jira] [Resolved] (SPARK-29158) Expose SerializableConfiguration for DSv2

2019-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29158. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25838

[jira] [Commented] (SPARK-29129) Flaky test: org.apache.spark.sql.hive.JavaDataFrameSuite

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934062#comment-16934062 ] Jungtaek Lim commented on SPARK-29129: -- SPARK-27538 seems to be similar, which is coupled with JDK

[jira] [Commented] (SPARK-29139) Flaky test: org.apache.spark.SparkContextSuite.test gpu driver resource files and discovery under local-cluster mode

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934034#comment-16934034 ] Jungtaek Lim commented on SPARK-29139: -- Looks like the build was super slow at that moment: |[test

[jira] [Updated] (SPARK-23197) Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23197: -- Component/s: Tests > Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle" >

[jira] [Commented] (SPARK-29162) Simplify NOT(isnull(x)) and NOT(isnotnull(x))

2019-09-19 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934032#comment-16934032 ] angerszhu commented on SPARK-29162: --- Rewrite in canonicalization is very useful. I will try to do

[jira] [Updated] (SPARK-27460) Running slowest test suites in their own forked JVMs for higher parallelism

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27460: -- Fix Version/s: 2.4.5 > Running slowest test suites in their own forked JVMs for higher

[jira] [Updated] (SPARK-27460) Running slowest test suites in their own forked JVMs for higher parallelism

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27460: -- Affects Version/s: 2.4.4 > Running slowest test suites in their own forked JVMs for higher

[jira] [Comment Edited] (SPARK-29130) Flay test: org.apache.spark.scheduler.BasicSchedulerIntegrationSuite."multi-stage job"

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934014#comment-16934014 ] Jungtaek Lim edited comment on SPARK-29130 at 9/20/19 3:41 AM: --- Actual

[jira] [Commented] (SPARK-29130) Flay test: org.apache.spark.scheduler.BasicSchedulerIntegrationSuite."multi-stage job"

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934014#comment-16934014 ] Jungtaek Lim commented on SPARK-29130: -- The root test failure might be "super simple job" as NPE

[jira] [Resolved] (SPARK-29134) Flaky test: org.apache.spark.sql.execution.SQLExecutionSuite.concurrent query execution with fork-join pool (SPARK-13747)

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29134. -- Resolution: Invalid Just affected by problematic commit (had been reverted) 

[jira] [Resolved] (SPARK-29133) Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcV1SchemaPruningSuite

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29133. -- Resolution: Invalid Just affected by problematic commit (had been reverted) 

[jira] [Resolved] (SPARK-29131) Flaky test: org.apache.spark.sql.execution.datasources.parquet.ParquetV1PartitionDiscoverySuite

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29131. -- Resolution: Invalid Just affected by problematic commit (had been reverted) 

[jira] [Resolved] (SPARK-29132) Flaky test: org.apache.spark.sql.execution.SQLJsonProtocolSuite.SparkListenerSQLExecutionEnd backward compatibility

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29132. -- Resolution: Invalid Just affected by problematic commit (had been reverted) 

[jira] [Commented] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2019-09-19 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933973#comment-16933973 ] Sean Owen commented on SPARK-28900: --- The meaning of JIRAs and umbrellas is already kind of nebulous,

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933967#comment-16933967 ] Dongjoon Hyun commented on SPARK-24417: --- Thanks, [~yumwang]! > Build and Run Spark on JDK11 >

[jira] [Commented] (SPARK-26003) Improve performance in SQLAppStatusListener

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933956#comment-16933956 ] Dongjoon Hyun commented on SPARK-26003: --- This is backported to `branch-2.4` via

[jira] [Resolved] (SPARK-29181) Cache preferred locations of checkpointed RDD

2019-09-19 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-29181. - Resolution: Duplicate > Cache preferred locations of checkpointed RDD >

[jira] [Assigned] (SPARK-29182) Cache preferred locations of checkpointed RDD

2019-09-19 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh reassigned SPARK-29182: --- Assignee: Liang-Chi Hsieh > Cache preferred locations of checkpointed RDD >

[jira] [Updated] (SPARK-26003) Improve performance in SQLAppStatusListener

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26003: -- Fix Version/s: 2.4.5 > Improve performance in SQLAppStatusListener >

[jira] [Commented] (SPARK-29181) Cache preferred locations of checkpointed RDD

2019-09-19 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933955#comment-16933955 ] Liang-Chi Hsieh commented on SPARK-29181: - [~dongjoon] Thanks. Not aware of creating duplicate

[jira] [Commented] (SPARK-29034) String Constants with C-style Escapes

2019-09-19 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933945#comment-16933945 ] jiaan.geng commented on SPARK-29034: Spark SQL has the treatment. {code:java} /** * Create a String

[jira] [Comment Edited] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933943#comment-16933943 ] Dongjoon Hyun edited comment on SPARK-28900 at 9/20/19 2:12 AM: I don't

[jira] [Commented] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933944#comment-16933944 ] Dongjoon Hyun commented on SPARK-28900: --- BTW, I'm running JDK11 build and testing including

[jira] [Commented] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933943#comment-16933943 ] Dongjoon Hyun commented on SPARK-28900: --- I don't think we can call it a day. My reason is that

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

2019-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933934#comment-16933934 ] Yuming Wang commented on SPARK-24417: - Yes. Please assign this umbrella JIRA issue to [~dbtsai]. >

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933933#comment-16933933 ] Hyukjin Kwon commented on SPARK-29102: -- Oh wait, but you tries to do it

[jira] [Resolved] (SPARK-29172) Fix some exception issue of explain commands

2019-09-19 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-29172. --- Resolution: Not A Problem > Fix some exception issue of explain commands >

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933930#comment-16933930 ] Hyukjin Kwon commented on SPARK-29102: -- Oh, right. Seems like regular Gzip should be explicitly

[jira] [Commented] (SPARK-29181) Cache preferred locations of checkpointed RDD

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933926#comment-16933926 ] Dongjoon Hyun commented on SPARK-29181: --- Hi, [~viirya]. Is this a duplicate of SPARK-29182? Then,

[jira] [Comment Edited] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933923#comment-16933923 ] Dongjoon Hyun edited comment on SPARK-28358 at 9/20/19 1:50 AM: It turns

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Description: The some python tests fail with `error: release unlocked lock`. -

[jira] [Commented] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933923#comment-16933923 ] Dongjoon Hyun commented on SPARK-28358: --- It turns out that this starts to make `master` SBT

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Description: The some python tests fail with `error: release unlocked lock`. -

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Description: The some python tests fail with `error: release unlocked lock`. -

[jira] [Updated] (SPARK-28358) Fix Flaky Tests - test_time_with_timezone (pyspark.sql.tests.test_serde.SerdeTests)

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28358: -- Description: The some python tests fail with `error: release unlocked lock`. -

[jira] [Commented] (SPARK-29136) Flaky test: Kafka test suites - metadata not propagated

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933909#comment-16933909 ] Jungtaek Lim commented on SPARK-29136: -- It would be better if we can reduce the chance of failures,

[jira] [Commented] (SPARK-29183) Upgrade JDK 11 Installation to 11.0.4

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933904#comment-16933904 ] Dongjoon Hyun commented on SPARK-29183: --- Thank you for the comments, [~shaneknapp] and [~srowen].

[jira] [Created] (SPARK-29185) Add new SaveMode types for Spark SQL jdbc datasource

2019-09-19 Thread Timothy Zhang (Jira)
Timothy Zhang created SPARK-29185: - Summary: Add new SaveMode types for Spark SQL jdbc datasource Key: SPARK-29185 URL: https://issues.apache.org/jira/browse/SPARK-29185 Project: Spark Issue

[jira] [Commented] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2019-09-19 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933892#comment-16933892 ] Sean Owen commented on SPARK-28900: --- Meh, I think it's kind of important to declare JDK 11 support

[jira] [Commented] (SPARK-29183) Upgrade JDK 11 Installation to 11.0.4

2019-09-19 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933885#comment-16933885 ] Sean Owen commented on SPARK-29183: --- I fully support Shane doing all the work to get this done! >

[jira] [Created] (SPARK-29184) I have installed pyspark and created code in notebook but when I am running it.its throwing error-Py4JJavaError: An error occurred while calling z:org.apache.spark.api.p

2019-09-19 Thread Asha Tambulker (Jira)
Asha Tambulker created SPARK-29184: -- Summary: I have installed pyspark and created code in notebook but when I am running it.its throwing error-Py4JJavaError: An error occurred while calling

[jira] [Commented] (SPARK-29183) Upgrade JDK 11 Installation to 11.0.4

2019-09-19 Thread shane knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933840#comment-16933840 ] shane knapp commented on SPARK-29183: - good point. i won't be able to get started until the middle

[jira] [Commented] (SPARK-29101) CSV datasource returns incorrect .count() from file with malformed records

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933822#comment-16933822 ] Dongjoon Hyun commented on SPARK-29101: --- This is backported to branch-2.4 for Apache Spark 2.4.5

[jira] [Updated] (SPARK-29101) CSV datasource returns incorrect .count() from file with malformed records

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29101: -- Labels: correctness (was: ) > CSV datasource returns incorrect .count() from file with

[jira] [Updated] (SPARK-29101) CSV datasource returns incorrect .count() from file with malformed records

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29101: -- Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Updated] (SPARK-29101) CSV datasource returns incorrect .count() from file with malformed records

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29101: -- Fix Version/s: 2.4.5 > CSV datasource returns incorrect .count() from file with malformed

[jira] [Commented] (SPARK-29183) Upgrade JDK 11 Installation to 11.0.4

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933815#comment-16933815 ] Dongjoon Hyun commented on SPARK-29183: --- cc [~shaneknapp] and [~srowen] > Upgrade JDK 11

[jira] [Created] (SPARK-29183) Upgrade JDK 11 Installation to 11.0.4

2019-09-19 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29183: - Summary: Upgrade JDK 11 Installation to 11.0.4 Key: SPARK-29183 URL: https://issues.apache.org/jira/browse/SPARK-29183 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-19 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933803#comment-16933803 ] Nicholas Chammas commented on SPARK-29102: -- [~hyukjin.kwon] - Would you happen to know how to

[jira] [Updated] (SPARK-28208) Upgrade to ORC 1.5.6

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28208: -- Component/s: Build > Upgrade to ORC 1.5.6 > > > Key:

[jira] [Updated] (SPARK-28208) When upgrading to ORC 1.5.6, the reader needs to be closed.

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28208: -- Issue Type: Improvement (was: Bug) > When upgrading to ORC 1.5.6, the reader needs to be

[jira] [Updated] (SPARK-28208) Upgrade to ORC 1.5.6

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28208: -- Summary: Upgrade to ORC 1.5.6 (was: When upgrading to ORC 1.5.6, the reader needs to be

[jira] [Commented] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933776#comment-16933776 ] Dongjoon Hyun commented on SPARK-28900: --- Can we move this out from the umbrella? > Test Pyspark,

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933746#comment-16933746 ] Dongjoon Hyun commented on SPARK-24417: --- Thanks! > Build and Run Spark on JDK11 >

[jira] [Assigned] (SPARK-24417) Build and Run Spark on JDK11

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-24417: - Assignee: DB Tsai > Build and Run Spark on JDK11 > > >

[jira] [Resolved] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

2019-09-19 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-28612. - Fix Version/s: 3.0.0 Assignee: Ryan Blue Resolution: Done Remerged in 

[jira] [Resolved] (SPARK-28822) Document USE DATABASE in SQL Reference

2019-09-19 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28822. - Fix Version/s: 3.0.0 Assignee: Shivu Sondur Resolution: Fixed > Document USE DATABASE

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

2019-09-19 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933700#comment-16933700 ] Sean Owen commented on SPARK-24417: --- I don't care at all, myself, as only a minor contributor. The

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933682#comment-16933682 ] Dongjoon Hyun commented on SPARK-24417: --- Hi, [~srowen], [~yumwang], [~hyukjin.kwon]. If you don't

[jira] [Resolved] (SPARK-29165) Set log level of log generated code as ERROR in case of compile error on generated code in UT

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29165. --- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by

[jira] [Assigned] (SPARK-29165) Set log level of log generated code as ERROR in case of compile error on generated code in UT

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29165: - Assignee: Jungtaek Lim > Set log level of log generated code as ERROR in case of

[jira] [Updated] (SPARK-28919) Add more profiles for JDK8/11 build test

2019-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28919: -- Parent: SPARK-24417 Issue Type: Sub-task (was: Improvement) > Add more profiles for

[jira] [Created] (SPARK-29182) Cache preferred locations of checkpointed RDD

2019-09-19 Thread Liang-Chi Hsieh (Jira)
Liang-Chi Hsieh created SPARK-29182: --- Summary: Cache preferred locations of checkpointed RDD Key: SPARK-29182 URL: https://issues.apache.org/jira/browse/SPARK-29182 Project: Spark Issue

[jira] [Resolved] (SPARK-29123) DecimalType multiplication precision loss

2019-09-19 Thread Benny Lu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benny Lu resolved SPARK-29123. -- Resolution: Information Provided > DecimalType multiplication precision loss >

[jira] [Created] (SPARK-29181) Cache preferred locations of checkpointed RDD

2019-09-19 Thread Liang-Chi Hsieh (Jira)
Liang-Chi Hsieh created SPARK-29181: --- Summary: Cache preferred locations of checkpointed RDD Key: SPARK-29181 URL: https://issues.apache.org/jira/browse/SPARK-29181 Project: Spark Issue

[jira] [Commented] (SPARK-29123) DecimalType multiplication precision loss

2019-09-19 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933560#comment-16933560 ] Marco Gaido commented on SPARK-29123: - [~benny] the point here is: Spark can represent decimals with

[jira] [Updated] (SPARK-29164) Rewrite coalesce(boolean, booleanLit) as boolean expression

2019-09-19 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-29164: --- Description: I propose the following expression rewrite optimizations: {code:java} coalesce(x:

[jira] [Updated] (SPARK-29164) Rewrite coalesce(boolean, booleanLit) as boolean expression

2019-09-19 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-29164: --- Description: I propose the following expression rewrite optimizations: {code:java} coalesce(x:

[jira] [Commented] (SPARK-29123) DecimalType multiplication precision loss

2019-09-19 Thread Benny Lu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933542#comment-16933542 ] Benny Lu commented on SPARK-29123: -- Thanks [~mgaido], that returns correct value. Do you know where

[jira] [Commented] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-09-19 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933539#comment-16933539 ] angerszhu commented on SPARK-29145: --- [~caidezhi655] Make a  PR for support  it, But I don't know if

[jira] [Commented] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-09-19 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933511#comment-16933511 ] angerszhu commented on SPARK-29145: --- Currently, IN/EXISTS/JOIN predicate sub-queries can only be used

[jira] [Commented] (SPARK-29143) Pyspark feature models support column setters/getters

2019-09-19 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933453#comment-16933453 ] Huaxin Gao commented on SPARK-29143: I will work on this. Thanks! > Pyspark feature models support

[jira] [Resolved] (SPARK-28985) Pyspark ClassificationModel and RegressionModel support column setters/getters/predict

2019-09-19 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28985. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25776

[jira] [Assigned] (SPARK-28985) Pyspark ClassificationModel and RegressionModel support column setters/getters/predict

2019-09-19 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28985: - Assignee: Huaxin Gao > Pyspark ClassificationModel and RegressionModel support column >

[jira] [Commented] (SPARK-25903) Flaky test: BarrierTaskContextSuite.throw exception on barrier() call timeout

2019-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1695#comment-1695 ] Jungtaek Lim commented on SPARK-25903: -- Hit this again: 

[jira] [Commented] (SPARK-29078) Spark shell fails if read permission is not granted to hive warehouse directory

2019-09-19 Thread Mihaly Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1692#comment-1692 ] Mihaly Toth commented on SPARK-29078: - I get AccessControlException from Hive pointing to

[jira] [Commented] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-19 Thread Dmitry Parfenchik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933322#comment-16933322 ] Dmitry Parfenchik commented on SPARK-29051: --- I can confirm, the original goal of [PR

[jira] [Comment Edited] (SPARK-29051) Spark Application UI search is not working for some fields

2019-09-19 Thread Dmitry Parfenchik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933322#comment-16933322 ] Dmitry Parfenchik edited comment on SPARK-29051 at 9/19/19 12:16 PM: -

[jira] [Commented] (SPARK-29034) String Constants with C-style Escapes

2019-09-19 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933296#comment-16933296 ] jiaan.geng commented on SPARK-29034: There are some tests in my production environment. {code:java}

[jira] [Commented] (SPARK-29034) String Constants with C-style Escapes

2019-09-19 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933295#comment-16933295 ] jiaan.geng commented on SPARK-29034: If set 

[jira] [Updated] (SPARK-29180) drop database throws Exception

2019-09-19 Thread ABHISHEK KUMAR GUPTA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-29180: - Attachment: DROP_DATABASE_Exception.png > drop database throws Exception >

[jira] [Commented] (SPARK-29180) drop database throws Exception

2019-09-19 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933269#comment-16933269 ] Sandeep Katta commented on SPARK-29180: --- I have found the reason, soon I will raise PR for this >

[jira] [Created] (SPARK-29180) drop database throws Exception

2019-09-19 Thread ABHISHEK KUMAR GUPTA (Jira)
ABHISHEK KUMAR GUPTA created SPARK-29180: Summary: drop database throws Exception Key: SPARK-29180 URL: https://issues.apache.org/jira/browse/SPARK-29180 Project: Spark Issue Type:

[jira] [Updated] (SPARK-29179) Aggregated Metrics by Executor table has refresh issue

2019-09-19 Thread ABHISHEK KUMAR GUPTA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-29179: - Attachment: AggregatedMetricsRefresh Issue2.png > Aggregated Metrics by

[jira] [Updated] (SPARK-29179) Aggregated Metrics by Executor table has refresh issue

2019-09-19 Thread ABHISHEK KUMAR GUPTA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-29179: - Attachment: AggregatedMetricsRefresh Issue1.png > Aggregated Metrics by

[jira] [Commented] (SPARK-29174) LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source

2019-09-19 Thread Ajith S (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933203#comment-16933203 ] Ajith S commented on SPARK-29174: - Had checked with original author for this

[jira] [Commented] (SPARK-29179) Aggregated Metrics by Executor table has refresh issue

2019-09-19 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933198#comment-16933198 ] Aman Omer commented on SPARK-29179: --- Working on this. > Aggregated Metrics by Executor table has

[jira] [Created] (SPARK-29179) Aggregated Metrics by Executor table has refresh issue

2019-09-19 Thread ABHISHEK KUMAR GUPTA (Jira)
ABHISHEK KUMAR GUPTA created SPARK-29179: Summary: Aggregated Metrics by Executor table has refresh issue Key: SPARK-29179 URL: https://issues.apache.org/jira/browse/SPARK-29179 Project: Spark

[jira] [Updated] (SPARK-29178) Improve shown message on sql show()

2019-09-19 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29178: Description: Add to show total rows with Date Frame table in

[jira] [Updated] (SPARK-29178) Improve shown message on sql show()

2019-09-19 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29178: Description: Add to show total rows with Date Frame table in

[jira] [Created] (SPARK-29178) Improve shown message on sql show()

2019-09-19 Thread Tomoko Komiyama (Jira)
Tomoko Komiyama created SPARK-29178: --- Summary: Improve shown message on sql show() Key: SPARK-29178 URL: https://issues.apache.org/jira/browse/SPARK-29178 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29174) LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source

2019-09-19 Thread daile (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933194#comment-16933194 ] daile commented on SPARK-29174: --- I tried to remove the judgment and found that it works. I don’t know if

[jira] [Updated] (SPARK-29173) Benchmark JDK 11 performance with FilterPushdownBenchmark

2019-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29173: Description: I was comparing the performance of JDK 8 and JDK 11 using

[jira] [Updated] (SPARK-29177) Zombie tasks prevents executor from releasing when task exceeds maxResultSize

2019-09-19 Thread Adrian Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Wang updated SPARK-29177: Description: When we fetch results from executors and found the total size has exceeded the

[jira] [Issue Comment Deleted] (SPARK-29174) LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source

2019-09-19 Thread daile (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] daile updated SPARK-29174: -- Comment: was deleted (was: /** * * Expected format: * {{{ * INSERT OVERWRITE DIRECTORY * [path] *

[jira] [Commented] (SPARK-29174) LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source

2019-09-19 Thread daile (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933181#comment-16933181 ] daile commented on SPARK-29174: --- /** * * Expected format: * {{{ * INSERT OVERWRITE DIRECTORY * [path]

[jira] [Created] (SPARK-29177) Zombie tasks prevents executor from releasing when task exceeds maxResultSize

2019-09-19 Thread Adrian Wang (Jira)
Adrian Wang created SPARK-29177: --- Summary: Zombie tasks prevents executor from releasing when task exceeds maxResultSize Key: SPARK-29177 URL: https://issues.apache.org/jira/browse/SPARK-29177 Project:

[jira] [Comment Edited] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-09-19 Thread Dezhi Cai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933174#comment-16933174 ] Dezhi Cai edited comment on SPARK-29145 at 9/19/19 8:38 AM: [~angerszhuuu]

[jira] [Commented] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-09-19 Thread Dezhi Cai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933174#comment-16933174 ] Dezhi Cai commented on SPARK-29145: --- [~angerszhuuu] we have workarounds,but using "on" when "join"

[jira] [Closed] (SPARK-27308) PropertyGraph Cypher query API (Python)

2019-09-19 Thread Martin Junghanns (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns closed SPARK-27308. > PropertyGraph Cypher query API (Python) > --- > >

  1   2   >