[jira] [Updated] (SPARK-32021) make_interval does not accept seconds >100

2020-06-18 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliusz Sompolski updated SPARK-32021: -- Description: In make_interval(years, months, weeks, days, hours, mins, secs), secs

[jira] [Created] (SPARK-32023) Generate spark streaming test jar with maven plugin maven-jar-plugin

2020-06-18 Thread Wan Kun (Jira)
Wan Kun created SPARK-32023: --- Summary: Generate spark streaming test jar with maven plugin maven-jar-plugin Key: SPARK-32023 URL: https://issues.apache.org/jira/browse/SPARK-32023 Project: Spark

[jira] [Commented] (SPARK-29679) Make interval type camparable and orderable

2020-06-18 Thread Bart Samwel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139339#comment-17139339 ] Bart Samwel commented on SPARK-29679: - I think there is a way to make this work in a reasonable way,

[jira] [Commented] (SPARK-32023) Generate spark streaming test jar with maven plugin maven-jar-plugin

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139385#comment-17139385 ] Apache Spark commented on SPARK-32023: -- User 'wankunde' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32023) Generate spark streaming test jar with maven plugin maven-jar-plugin

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32023: Assignee: Apache Spark > Generate spark streaming test jar with maven plugin

[jira] [Assigned] (SPARK-32023) Generate spark streaming test jar with maven plugin maven-jar-plugin

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32023: Assignee: (was: Apache Spark) > Generate spark streaming test jar with maven plugin

[jira] [Created] (SPARK-32022) Can many executors share one gpu for spark3.0?

2020-06-18 Thread liucheng (Jira)
liucheng created SPARK-32022: Summary: Can many executors share one gpu for spark3.0? Key: SPARK-32022 URL: https://issues.apache.org/jira/browse/SPARK-32022 Project: Spark Issue Type: Question

[jira] [Created] (SPARK-32021) make_interval does not accept seconds >100

2020-06-18 Thread Juliusz Sompolski (Jira)
Juliusz Sompolski created SPARK-32021: - Summary: make_interval does not accept seconds >100 Key: SPARK-32021 URL: https://issues.apache.org/jira/browse/SPARK-32021 Project: Spark Issue

[jira] [Commented] (SPARK-32002) spark error while select nest data

2020-06-18 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139249#comment-17139249 ] angerszhu commented on SPARK-32002: --- for B there is tow level ArrayType > spark error while select

[jira] [Comment Edited] (SPARK-32002) spark error while select nest data

2020-06-18 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139249#comment-17139249 ] angerszhu edited comment on SPARK-32002 at 6/18/20, 9:17 AM: - for B there is

[jira] [Commented] (SPARK-31982) Spark sequence doesn't handle date increments that cross DST

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139313#comment-17139313 ] Apache Spark commented on SPARK-31982: -- User 'TJX2014' has created a pull request for this issue:

[jira] [Commented] (SPARK-32023) Generate spark streaming test jar with maven plugin maven-jar-plugin

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139386#comment-17139386 ] Apache Spark commented on SPARK-32023: -- User 'wankunde' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31982) Spark sequence doesn't handle date increments that cross DST

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31982: Assignee: (was: Apache Spark) > Spark sequence doesn't handle date increments that

[jira] [Assigned] (SPARK-31982) Spark sequence doesn't handle date increments that cross DST

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31982: Assignee: Apache Spark > Spark sequence doesn't handle date increments that cross DST >

[jira] [Commented] (SPARK-31982) Spark sequence doesn't handle date increments that cross DST

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139311#comment-17139311 ] Apache Spark commented on SPARK-31982: -- User 'TJX2014' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-32002) spark error while select nest data

2020-06-18 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139249#comment-17139249 ] angerszhu edited comment on SPARK-32002 at 6/18/20, 9:39 AM: - this is

[jira] [Commented] (SPARK-24163) Support "ANY" or sub-query for Pivot "IN" clause

2020-06-18 Thread Peter Wicks (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139474#comment-17139474 ] Peter Wicks commented on SPARK-24163: - [~maryannxue] I don't think this work was completed.  Can the

[jira] [Assigned] (SPARK-32002) spark error while select nest data

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32002: Assignee: Apache Spark > spark error while select nest data >

[jira] [Commented] (SPARK-32002) spark error while select nest data

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139464#comment-17139464 ] Apache Spark commented on SPARK-32002: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-32002) spark error while select nest data

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32002: Assignee: (was: Apache Spark) > spark error while select nest data >

[jira] [Created] (SPARK-32024) Disk usage tracker went negative in HistoryServerDiskManager

2020-06-18 Thread Zhen Li (Jira)
Zhen Li created SPARK-32024: --- Summary: Disk usage tracker went negative in HistoryServerDiskManager Key: SPARK-32024 URL: https://issues.apache.org/jira/browse/SPARK-32024 Project: Spark Issue

[jira] [Assigned] (SPARK-7101) Spark SQL should support java.sql.Time

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-7101: --- Assignee: Apache Spark > Spark SQL should support java.sql.Time >

[jira] [Assigned] (SPARK-7101) Spark SQL should support java.sql.Time

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-7101: --- Assignee: (was: Apache Spark) > Spark SQL should support java.sql.Time >

[jira] [Commented] (SPARK-7101) Spark SQL should support java.sql.Time

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139417#comment-17139417 ] Apache Spark commented on SPARK-7101: - User 'younggyuchun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31336) Support Oracle Kerberos login in JDBC connector

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31336: Assignee: Apache Spark > Support Oracle Kerberos login in JDBC connector >

[jira] [Commented] (SPARK-31336) Support Oracle Kerberos login in JDBC connector

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139515#comment-17139515 ] Apache Spark commented on SPARK-31336: -- User 'gaborgsomogyi' has created a pull request for this

[jira] [Assigned] (SPARK-31336) Support Oracle Kerberos login in JDBC connector

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31336: Assignee: (was: Apache Spark) > Support Oracle Kerberos login in JDBC connector >

[jira] [Assigned] (SPARK-31960) Only populate Hadoop classpath for no-hadoop build

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31960: - Assignee: DB Tsai (was: Apache Spark) > Only populate Hadoop classpath for no-hadoop

[jira] [Resolved] (SPARK-32020) Refactor the logic to compute SPARK_HOME into a common place.

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32020. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32020) Refactor the logic to compute SPARK_HOME into a common place.

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32020: Assignee: Dilip Biswal > Refactor the logic to compute SPARK_HOME into a common place. >

[jira] [Commented] (SPARK-32024) Disk usage tracker went negative in HistoryServerDiskManager

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139455#comment-17139455 ] Apache Spark commented on SPARK-32024: -- User 'zhli1142015' has created a pull request for this

[jira] [Assigned] (SPARK-32024) Disk usage tracker went negative in HistoryServerDiskManager

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32024: Assignee: Apache Spark > Disk usage tracker went negative in HistoryServerDiskManager >

[jira] [Commented] (SPARK-32024) Disk usage tracker went negative in HistoryServerDiskManager

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139456#comment-17139456 ] Apache Spark commented on SPARK-32024: -- User 'zhli1142015' has created a pull request for this

[jira] [Assigned] (SPARK-32024) Disk usage tracker went negative in HistoryServerDiskManager

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32024: Assignee: (was: Apache Spark) > Disk usage tracker went negative in

[jira] [Commented] (SPARK-29679) Make interval type camparable and orderable

2020-06-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139870#comment-17139870 ] Wenchen Fan commented on SPARK-29679: - We can normalize the days, but what about the months? Shall

[jira] [Commented] (SPARK-19256) Hive bucketing support

2020-06-18 Thread Virendhar Sivaraman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139931#comment-17139931 ] Virendhar Sivaraman commented on SPARK-19256: - This is a much needed feature +1 vote > Hive

[jira] [Commented] (SPARK-26494) Support Oracle TIMESTAMP WITH LOCAL TIME ZONE type

2020-06-18 Thread Jeff Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139930#comment-17139930 ] Jeff Evans commented on SPARK-26494: The pull request was closed as stale. Can someone please

[jira] [Commented] (SPARK-19355) Use map output statistices to improve global limit's parallelism

2020-06-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140071#comment-17140071 ] L. C. Hsieh commented on SPARK-19355: - I think this can be implemented now on top of AQE. But the

[jira] [Created] (SPARK-32026) Support consistency on Prometheus driver and executor metrics format

2020-06-18 Thread Eren Avsarogullari (Jira)
Eren Avsarogullari created SPARK-32026: -- Summary: Support consistency on Prometheus driver and executor metrics format Key: SPARK-32026 URL: https://issues.apache.org/jira/browse/SPARK-32026

[jira] [Commented] (SPARK-32026) Support consistency on Prometheus driver and executor metrics format

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140076#comment-17140076 ] Apache Spark commented on SPARK-32026: -- User 'erenavsarogullari' has created a pull request for

[jira] [Updated] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

2020-06-18 Thread Tathagata Das (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-32017: -- Fix Version/s: (was: 3.0.1) > Make Pyspark Hadoop 3.2+ Variant available in PyPI >

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24266: -- Target Version/s: 3.1.0, 2.4.7 (was: 3.0.0, 3.1.0, 2.4.7) > Spark client terminates while

[jira] [Assigned] (SPARK-29194) JDK11 QA

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29194: - Assignee: Dongjoon Hyun > JDK11 QA > > > Key: SPARK-29194 >

[jira] [Resolved] (SPARK-29194) JDK11 QA

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29194. --- Resolution: Done > JDK11 QA > > > Key: SPARK-29194 >

[jira] [Reopened] (SPARK-29194) JDK11 QA

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-29194: --- > JDK11 QA > > > Key: SPARK-29194 > URL:

[jira] [Comment Edited] (SPARK-32022) Can many executors share one gpu for spark3.0?

2020-06-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140072#comment-17140072 ] L. C. Hsieh edited comment on SPARK-32022 at 6/19/20, 12:34 AM: Based on

[jira] [Commented] (SPARK-32022) Can many executors share one gpu for spark3.0?

2020-06-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140072#comment-17140072 ] L. C. Hsieh commented on SPARK-32022: - Based on your description, it sounds more like

[jira] [Comment Edited] (SPARK-32022) Can many executors share one gpu for spark3.0?

2020-06-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140072#comment-17140072 ] L. C. Hsieh edited comment on SPARK-32022 at 6/19/20, 12:34 AM: Based on

[jira] [Issue Comment Deleted] (SPARK-20628) Keep track of nodes which are going to be shut down & avoid scheduling new tasks

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-20628: - Comment: was deleted (was: [~holden] Looks like it had to be a SPIP. Do we have a design doc

[jira] [Updated] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32017: - Priority: Major (was: Minor) > Make Pyspark Hadoop 3.2+ Variant available in PyPI >

[jira] [Updated] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28900: -- Target Version/s: (was: 3.0.0) > Test Pyspark, SparkR on JDK 11 with run-tests >

[jira] [Commented] (SPARK-29194) JDK11 QA

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140070#comment-17140070 ] Dongjoon Hyun commented on SPARK-29194: --- 3.0.0 is released. For the future JIRAs, we will consider

[jira] [Updated] (SPARK-28900) Test Pyspark, SparkR on JDK 11 with run-tests

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28900: -- Parent: (was: SPARK-29194) Issue Type: Improvement (was: Sub-task) > Test

[jira] [Resolved] (SPARK-29194) JDK11 QA

2020-06-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29194. --- Resolution: Done > JDK11 QA > > > Key: SPARK-29194 >

[jira] [Comment Edited] (SPARK-32022) Can many executors share one gpu for spark3.0?

2020-06-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140072#comment-17140072 ] L. C. Hsieh edited comment on SPARK-32022 at 6/19/20, 12:36 AM: Based on

[jira] [Assigned] (SPARK-32026) Support consistency on Prometheus driver and executor metrics format

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32026: Assignee: Apache Spark > Support consistency on Prometheus driver and executor metrics

[jira] [Assigned] (SPARK-32026) Support consistency on Prometheus driver and executor metrics format

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32026: Assignee: (was: Apache Spark) > Support consistency on Prometheus driver and

[jira] [Commented] (SPARK-20624) Add better handling for node shutdown

2020-06-18 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140094#comment-17140094 ] Holden Karau commented on SPARK-20624: -- [~hyukjin.kwon] the design doc is at

[jira] [Updated] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32017: - Target Version/s: (was: 3.0.0) > Make Pyspark Hadoop 3.2+ Variant available in PyPI >

[jira] [Created] (SPARK-32025) CSV schema inference with boolean & integer

2020-06-18 Thread Brian Wallace (Jira)
Brian Wallace created SPARK-32025: - Summary: CSV schema inference with boolean & integer Key: SPARK-32025 URL: https://issues.apache.org/jira/browse/SPARK-32025 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24266) Spark client terminates while driver is still running

2020-06-18 Thread Jim Kleckner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140028#comment-17140028 ] Jim Kleckner commented on SPARK-24266: -- +1 for this patch set if it reviews cleanly. > Spark

[jira] [Commented] (SPARK-32004) Drop references to slave

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139893#comment-17139893 ] Apache Spark commented on SPARK-32004: -- User 'holdenk' has created a pull request for this issue:

[jira] [Commented] (SPARK-32004) Drop references to slave

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139892#comment-17139892 ] Apache Spark commented on SPARK-32004: -- User 'holdenk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32004) Drop references to slave

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32004: Assignee: (was: Apache Spark) > Drop references to slave >

[jira] [Assigned] (SPARK-32004) Drop references to slave

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32004: Assignee: Apache Spark > Drop references to slave > > >

[jira] [Updated] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

2020-06-18 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-32017: Target Version/s: 3.1.0 > Make Pyspark Hadoop 3.2+ Variant available in PyPI >

[jira] [Updated] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure

2020-06-18 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-31845: --- Summary: DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure (was: DAGSchedulerSuite:

[jira] [Created] (SPARK-32027) EventLoggingListener threw java.util.ConcurrentModificationException

2020-06-18 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-32027: --- Summary: EventLoggingListener threw java.util.ConcurrentModificationException Key: SPARK-32027 URL: https://issues.apache.org/jira/browse/SPARK-32027 Project: Spark

[jira] [Assigned] (SPARK-30865) Refactor DateTimeUtils

2020-06-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30865: --- Assignee: Maxim Gekk > Refactor DateTimeUtils > -- > >

[jira] [Resolved] (SPARK-30865) Refactor DateTimeUtils

2020-06-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30865. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27617

[jira] [Commented] (SPARK-20624) Add better handling for node shutdown

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140103#comment-17140103 ] Hyukjin Kwon commented on SPARK-20624: -- I understand the JIRA and design doc were made earlier.

[jira] [Commented] (SPARK-20624) Add better handling for node shutdown

2020-06-18 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140124#comment-17140124 ] Holden Karau commented on SPARK-20624: -- [~hyukjin.kwon] that doesn't seem comparable, the history

[jira] [Updated] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and support checkAnswer

2020-06-18 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-31845: --- Summary: DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and support checkAnswer (was:

[jira] [Commented] (SPARK-32022) Can many executors share one gpu for spark3.0?

2020-06-18 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140187#comment-17140187 ] L. C. Hsieh commented on SPARK-32022: - I don't think spark.executor.resource.gpu.amount supports

[jira] [Created] (SPARK-32029) Make activeSession null when application end

2020-06-18 Thread ulysses you (Jira)
ulysses you created SPARK-32029: --- Summary: Make activeSession null when application end Key: SPARK-32029 URL: https://issues.apache.org/jira/browse/SPARK-32029 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-31887) Date casting to string is giving wrong value

2020-06-18 Thread Amit Gupta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140127#comment-17140127 ] Amit Gupta commented on SPARK-31887: This issue is fixed in spark 3.0. Spark 2.5 and 2.6 are

[jira] [Commented] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140144#comment-17140144 ] Apache Spark commented on SPARK-31845: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140143#comment-17140143 ] Apache Spark commented on SPARK-31845: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31845: Assignee: (was: Apache Spark) > DAGSchedulerSuite: Reuse

[jira] [Assigned] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31845: Assignee: Apache Spark > DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure >

[jira] [Commented] (SPARK-27148) Support CURRENT_TIME and LOCALTIME when ANSI mode enabled

2020-06-18 Thread YoungGyu Chun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140150#comment-17140150 ] YoungGyu Chun commented on SPARK-27148: --- I am working on SPARK-7101. After that, I will get this

[jira] [Updated] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and support checkAnswer

2020-06-18 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-31845: --- Description: DAGSchedulerSuite provides completeNextStageWithFetchFailure to make next stage

[jira] [Updated] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and support checkAnswer

2020-06-18 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-31845: --- Description: DAGSchedulerSuite provides completeNextStageWithFetchFailure to make next stage

[jira] [Updated] (SPARK-31845) DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and support checkAnswer

2020-06-18 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-31845: --- Description: *First* DAGSchedulerSuite provides completeNextStageWithFetchFailure to make next

[jira] [Commented] (SPARK-32022) Can many executors share one gpu for spark3.0?

2020-06-18 Thread liucheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140170#comment-17140170 ] liucheng commented on SPARK-32022: -- [~viirya] hi,I think you don't get my idea. I tested one executor

[jira] [Updated] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-18 Thread Zhen Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhen Li updated SPARK-32028: Attachment: wrong_attemptJPG.JPG multi_same.JPG > App id link in history summary page

[jira] [Created] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-18 Thread Zhen Li (Jira)
Zhen Li created SPARK-32028: --- Summary: App id link in history summary page point to wrong application attempt Key: SPARK-32028 URL: https://issues.apache.org/jira/browse/SPARK-32028 Project: Spark

[jira] [Updated] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-18 Thread Zhen Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhen Li updated SPARK-32028: Description: App id link in history summary page url is wrong, for multi attempts case. for details,

[jira] [Commented] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140217#comment-17140217 ] Apache Spark commented on SPARK-32028: -- User 'zhli1142015' has created a pull request for this

[jira] [Assigned] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32028: Assignee: (was: Apache Spark) > App id link in history summary page point to wrong

[jira] [Assigned] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32028: Assignee: Apache Spark > App id link in history summary page point to wrong application

[jira] [Commented] (SPARK-32028) App id link in history summary page point to wrong application attempt

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140218#comment-17140218 ] Apache Spark commented on SPARK-32028: -- User 'zhli1142015' has created a pull request for this

[jira] [Assigned] (SPARK-31894) Introduce UnsafeRow format validation for streaming state store

2020-06-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31894: --- Assignee: Yuanjian Li > Introduce UnsafeRow format validation for streaming state store >

[jira] [Resolved] (SPARK-31894) Introduce UnsafeRow format validation for streaming state store

2020-06-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31894. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28707

[jira] [Commented] (SPARK-31871) Display the canvas element icon for sorting column

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139186#comment-17139186 ] Apache Spark commented on SPARK-31871: -- User 'liucht-inspur' has created a pull request for this

[jira] [Commented] (SPARK-31871) Display the canvas element icon for sorting column

2020-06-18 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139189#comment-17139189 ] Apache Spark commented on SPARK-31871: -- User 'liucht-inspur' has created a pull request for this

[jira] [Resolved] (SPARK-31960) Only populate Hadoop classpath for no-hadoop build

2020-06-18 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-31960. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28788

[jira] [Resolved] (SPARK-27280) infer filters from Join's OR condition

2020-06-18 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-27280. - Resolution: Duplicate > infer filters from Join's OR condition >

[jira] [Commented] (SPARK-20628) Keep track of nodes which are going to be shut down & avoid scheduling new tasks

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139132#comment-17139132 ] Hyukjin Kwon commented on SPARK-20628: -- [~holden] Looks like it had to be a SPIP. Do we have a

[jira] [Commented] (SPARK-20624) Add better handling for node shutdown

2020-06-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139133#comment-17139133 ] Hyukjin Kwon commented on SPARK-20624: -- [~holden] Looks like it had to be a SPIP. Do we have a

  1   2   >