[jira] [Commented] (SPARK-30334) Add metadata around semi-structured columns to Spark

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840926#comment-17840926 ] Dongjoon Hyun commented on SPARK-30334: --- I removed the outdated target version from this issue

[jira] [Commented] (SPARK-24942) Improve cluster resource management with jobs containing barrier stage

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840913#comment-17840913 ] Dongjoon Hyun commented on SPARK-24942: --- I removed the outdated target version, `3.2.0`, from

[jira] [Updated] (SPARK-24942) Improve cluster resource management with jobs containing barrier stage

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24942: -- Target Version/s: (was: 3.2.0) > Improve cluster resource management with jobs contain

[jira] [Commented] (SPARK-44111) Prepare Apache Spark 4.0.0

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840853#comment-17840853 ] Dongjoon Hyun commented on SPARK-44111: --- Yes, we will provide `4.0.0-preview` in advance

[FYI] SPARK-47993: Drop Python 3.8

2024-04-25 Thread Dongjoon Hyun
FYI, there is a proposal to drop Python 3.8 because its EOL is October 2024. https://github.com/apache/spark/pull/46228 [SPARK-47993][PYTHON] Drop Python 3.8 Since it's still alive and there will be an overlap between the lifecycle of Python 3.8 and Apache Spark 4.0.0, please give us your

[jira] [Resolved] (SPARK-47987) Enable `ArrowParityTests.test_createDataFrame_empty_partition`

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47987. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46220 [https

[jira] [Resolved] (SPARK-47990) Upgrade `zstd-jni` to 1.5.6-3

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47990. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46225 [https

[jira] [Commented] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840644#comment-17840644 ] Dongjoon Hyun commented on SPARK-46122: --- I sent the discussion thread for this issue. - [https

[jira] [Resolved] (ORC-1704) Migration to Scala 2.13 of Apache Spark 3.5.1 at SparkBenchmark

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1704. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (ORC-1704) Migration to Scala 2.13 of Apache Spark 3.5.1 at SparkBenchmark

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1704: -- Assignee: dzcxzl > Migration to Scala 2.13 of Apache Spark 3.5.1 at SparkBenchm

[DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-24 Thread Dongjoon Hyun
Hi, All. It's great to see community activities to polish 4.0.0 more and more. Thank you all. I'd like to bring SPARK-46122 (another SQL topic) to you from the subtasks of SPARK-4 (Prepare Apache Spark 4.0.0), - https://issues.apache.org/jira/browse/SPARK-46122 Set

[jira] [Updated] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46122: -- Summary: Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default (was: Set

[jira] [Updated] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to false by default

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46122: -- Summary: Set `spark.sql.legacy.createHiveTableByDefault` to false by default (was: Disable

[jira] [Resolved] (SPARK-47979) Use Hive tables explicitly for Hive table capability tests

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47979. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46211 [https

[jira] [Created] (SPARK-47979) Use Hive table explicitly for Hive table capability tests

2024-04-24 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47979: - Summary: Use Hive table explicitly for Hive table capability tests Key: SPARK-47979 URL: https://issues.apache.org/jira/browse/SPARK-47979 Project: Spark

[jira] [Updated] (SPARK-47979) Use Hive tables explicitly for Hive table capability tests

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47979: -- Summary: Use Hive tables explicitly for Hive table capability tests (was: Use Hive table

[jira] [Assigned] (SPARK-45265) Support Hive 4.0 metastore

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45265: - Assignee: (was: Attila Zsolt Piros) > Support Hive 4.0 metast

[jira] [Updated] (SPARK-44677) Drop legacy Hive-based ORC file format

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44677: -- Parent: (was: SPARK-44111) Issue Type: Task (was: Sub-task) > Drop legacy H

[jira] [Commented] (SPARK-47499) Reuse `test_help_command` in Connect

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840514#comment-17840514 ] Dongjoon Hyun commented on SPARK-47499: --- Thank you for collecting this to the umbrella Jira

[jira] [Assigned] (SPARK-47633) Cache miss for queries using JOIN LATERAL with join condition

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47633: - Assignee: Bruce Robbins > Cache miss for queries using JOIN LATERAL with j

[jira] [Resolved] (SPARK-47633) Cache miss for queries using JOIN LATERAL with join condition

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47633. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46190 [https

Re: [VOTE] Apache Hive 2.3.10 Release Candidate 0

2024-04-24 Thread Dongjoon Hyun
Hi, Chao. The vote email seems to have a wrong repository link, 1106. IIUC, 1128 is the valid one, isn't it? Dongjoon. On 2024/04/20 20:01:53 Chao Sun wrote: > Apache Hive 2.3.10 Release Candidate 0 is available here: > https://people.apache.org/~sunchao/apache-hive-2.3.10-rc-0/ > Maven

[jira] [Resolved] (ORC-1703) Bump `maven-jar-plugin` to 3.4.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1703. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1905 [https

[jira] [Assigned] (ORC-1703) Bump `maven-jar-plugin` to 3.4.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1703: -- Assignee: Dongjoon Hyun > Bump `maven-jar-plugin` to 3.

[jira] [Created] (ORC-1703) Bump `maven-jar-plugin` to 3.4.1

2024-04-24 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1703: -- Summary: Bump `maven-jar-plugin` to 3.4.1 Key: ORC-1703 URL: https://issues.apache.org/jira/browse/ORC-1703 Project: ORC Issue Type: Bug Components

[jira] [Resolved] (ORC-1702) Bump `bcpkix-jdk18on` to 1.78.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1702. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1904 [https

[jira] [Assigned] (ORC-1702) Bump `bcpkix-jdk18on` to 1.78.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1702: -- Assignee: Dongjoon Hyun > Bump `bcpkix-jdk18on` to 1.7

[jira] [Assigned] (ORC-1701) Bump `threeten-extra` to 1.8.0

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1701: -- Assignee: Dongjoon Hyun > Bump `threeten-extra` to 1.

[jira] [Resolved] (ORC-1701) Bump `threeten-extra` to 1.8.0

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1701. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1903 [https

[jira] [Created] (ORC-1702) Bump `bcpkix-jdk18on` to 1.78.1

2024-04-24 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1702: -- Summary: Bump `bcpkix-jdk18on` to 1.78.1 Key: ORC-1702 URL: https://issues.apache.org/jira/browse/ORC-1702 Project: ORC Issue Type: Bug Components

[jira] [Created] (ORC-1701) Bump `threeten-extra` to 1.8.0

2024-04-24 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1701: -- Summary: Bump `threeten-extra` to 1.8.0 Key: ORC-1701 URL: https://issues.apache.org/jira/browse/ORC-1701 Project: ORC Issue Type: Improvement

[jira] [Resolved] (ORC-1699) Fix SparkBenchmark in Parquet format according to SPARK-40918

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1699. Fix Version/s: 2.0.1 1.9.4 2.1.0 Resolution: Fixed

[jira] [Assigned] (ORC-1699) Fix SparkBenchmark in Parquet format according to SPARK-40918

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1699: -- Assignee: dzcxzl > Fix SparkBenchmark in Parquet format according to SPARK-40

[jira] [Assigned] (ORC-1698) Upgrade `commons-cli` to 1.7.0

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1698: -- Assignee: dzcxzl > Upgrade `commons-cli` to 1.

[jira] [Resolved] (ORC-1698) Upgrade `commons-cli` to 1.7.0

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1698. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-47974) Remove install_scala from build/mvn

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47974: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Remove install_sc

[jira] [Resolved] (SPARK-47974) Remove install_scala from build/mvn

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47974. --- Fix Version/s: 4.0.0 Assignee: Cheng Pan Resolution: Fixed This is resolved

[jira] [Resolved] (SPARK-47969) Make `test_creation_index` deterministic

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47969. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46200 [https

[jira] [Resolved] (SPARK-47956) sanity check for unresolved LCA reference

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47956. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46185 [https

[jira] [Assigned] (SPARK-47956) sanity check for unresolved LCA reference

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47956: - Assignee: Wenchen Fan > sanity check for unresolved LCA refere

[jira] [Assigned] (SPARK-47948) Upgrade the minimum Pandas version to 2.0.0

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47948: - Assignee: Haejoon Lee > Upgrade the minimum Pandas version to 2.

[jira] [Resolved] (SPARK-47948) Upgrade the minimum Pandas version to 2.0.0

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47948. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46175 [https

[jira] [Updated] (SPARK-47948) Upgrade the minimum Pandas version to 2.0.0

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47948: -- Summary: Upgrade the minimum Pandas version to 2.0.0 (was: Bump Pandas to 2.0.0) > Upgr

[jira] [Resolved] (SPARK-47949) MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47949. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46176 [https

[jira] [Assigned] (SPARK-47949) MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47949: - Assignee: Kent Yao > MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubu

[jira] [Assigned] (SPARK-47953) MsSQLServer: Document Mapping Spark SQL Data Types to Microsoft SQL Server

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47953: - Assignee: Kent Yao > MsSQLServer: Document Mapping Spark SQL Data Types to Micros

[jira] [Resolved] (SPARK-47953) MsSQLServer: Document Mapping Spark SQL Data Types to Microsoft SQL Server

2024-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47953. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46177 [https

[jira] [Assigned] (SPARK-47943) Add Operator CI Task for Java Build and Test

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47943: - Assignee: Zhou JIANG > Add Operator CI Task for Java Build and T

[jira] [Resolved] (SPARK-47943) Add Operator CI Task for Java Build and Test

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47943. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 7 [https

[jira] [Resolved] (SPARK-47929) Setup Static Analysis for Operator

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47929. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 6 [https

[jira] [Resolved] (SPARK-47938) MsSQLServer: Cannot find data type BYTE error

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47938. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46164 [https

[jira] [Resolved] (SPARK-47937) Fix docstring of `hll_sketch_agg`

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47937. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46163 [https

[jira] [Assigned] (SPARK-47937) Fix docstring of `hll_sketch_agg`

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47937: - Assignee: Ruifeng Zheng > Fix docstring of `hll_sketch_

[jira] [Resolved] (SPARK-47904) Preserve case in Avro schema when using enableStableIdentifiersForUnionType

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47904. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46169 [https

[jira] [Updated] (SPARK-47904) Preserve case in Avro schema when using enableStableIdentifiersForUnionType

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47904: -- Fix Version/s: 4.0.0 > Preserve case in Avro schema when us

[jira] [Assigned] (SPARK-47904) Preserve case in Avro schema when using enableStableIdentifiersForUnionType

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47904: - Assignee: Ivan Sadikov > Preserve case in Avro schema when us

[jira] [Resolved] (SPARK-47942) Drop K8s v1.26 Support

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47942. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46168 [https

[jira] [Assigned] (SPARK-47942) Drop K8s v1.26 Support

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47942: - Assignee: Dongjoon Hyun > Drop K8s v1.26 Supp

[jira] [Created] (SPARK-47942) Drop K8s v1.26 Support

2024-04-22 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47942: - Summary: Drop K8s v1.26 Support Key: SPARK-47942 URL: https://issues.apache.org/jira/browse/SPARK-47942 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-47940) Upgrade `guava` dependency to `33.1.0-jre` in Docker IT

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47940: -- Reporter: Cheng Pan (was: Dongjoon Hyun) > Upgrade `guava` dependency to `33.1.0-

[jira] [Resolved] (SPARK-47940) Upgrade `guava` dependency to `33.1.0-jre` in Docker IT

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47940. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46167 [https

[jira] [Created] (SPARK-47940) Upgrade `guava` dependency to `33.1.0-jre` in Docker IT

2024-04-22 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47940: - Summary: Upgrade `guava` dependency to `33.1.0-jre` in Docker IT Key: SPARK-47940 URL: https://issues.apache.org/jira/browse/SPARK-47940 Project: Spark

[jira] [Assigned] (SPARK-47935) Pin pandas==2.0.3 for pypy3.8

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47935: - Assignee: Ruifeng Zheng > Pin pandas==2.0.3 for pypy

[jira] [Resolved] (SPARK-47935) Pin pandas==2.0.3 for pypy3.8

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47935. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46159 [https

[jira] [Resolved] (ORC-1696) Fix ClassCastException when reading avro decimal type in bechmark

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1696. Fix Version/s: 1.9.4 1.8.8 1.7.11 2.0.1

[jira] [Updated] (ORC-1696) Fix ClassCastException when reading avro decimal type in bechmark

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated ORC-1696: --- Issue Type: Bug (was: Improvement) > Fix ClassCastException when reading avro decimal t

[jira] [Assigned] (ORC-1696) Fix ClassCastException when reading avro decimal type in bechmark

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1696: -- Assignee: dzcxzl > Fix ClassCastException when reading avro decimal type in bechm

[jira] [Resolved] (SPARK-47930) Upgrade RoaringBitmap to 1.0.6

2024-04-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47930. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46152 [https

[FYI] SPARK-47046: Apache Spark 4.0.0 Dependency Audit and Cleanup

2024-04-21 Thread Dongjoon Hyun
on the above reports or have new ones for Apache Spark 4.0.0. Dongjoon Hyun

Re: [DISCUSS] Un-deprecate Trigger.Once

2024-04-19 Thread Dongjoon Hyun
For that case, I believe it's enough for us to revise the deprecation message only by making sure that Apache Spark will keep it without removal for backward-compatibility purposes only. That's what the users asked, isn't that? > deprecation of Trigger.Once confuses users that the trigger won't

[jira] [Resolved] (SPARK-47925) Mark `BloomFilterAggregateQuerySuite` as `ExtendedSQLTest`

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47925. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46145 [https

[jira] [Assigned] (SPARK-47925) Mark `BloomFilterAggregateQuerySuite` as `ExtendedSQLTest`

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47925: - Assignee: Dongjoon Hyun > Mark `BloomFilterAggregateQuerySuite` as `ExtendedSQLT

[jira] [Resolved] (SPARK-47924) Add a debug log to `DiskStore.moveFileToBlock`

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47924. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46144 [https

[jira] [Assigned] (SPARK-47924) Add a debug log to `DiskStore.moveFileToBlock`

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47924: - Assignee: Dongjoon Hyun > Add a debug log to `DiskStore.moveFileToBl

[jira] [Resolved] (SPARK-47923) Upgrade the minimum version of `arrow` R package to 10.0.0

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47923. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46142 [https

[jira] [Created] (SPARK-47925) Mark `BloomFilterAggregateQuerySuite` as `ExtendedSQLTest`

2024-04-19 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47925: - Summary: Mark `BloomFilterAggregateQuerySuite` as `ExtendedSQLTest` Key: SPARK-47925 URL: https://issues.apache.org/jira/browse/SPARK-47925 Project: Spark

[jira] [Created] (SPARK-47924) Add a debug log to `DiskStore.moveFileToBlock`

2024-04-19 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47924: - Summary: Add a debug log to `DiskStore.moveFileToBlock` Key: SPARK-47924 URL: https://issues.apache.org/jira/browse/SPARK-47924 Project: Spark Issue Type

[jira] [Updated] (SPARK-47737) Upgrade the minimum version of PyArrow to 10.0.0

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47737: -- Summary: Upgrade the minimum version of PyArrow to 10.0.0 (was: Bump PyArrow to 10.0.0

[jira] [Assigned] (SPARK-47923) Upgrade the minimum version of `arrow` R package to 10.0.0

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47923: - Assignee: Dongjoon Hyun > Upgrade the minimum version of `arrow` R package to 10.

[jira] [Created] (SPARK-47923) Upgrade the minimum version of `arrow` R package to 10.0.0

2024-04-19 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47923: - Summary: Upgrade the minimum version of `arrow` R package to 10.0.0 Key: SPARK-47923 URL: https://issues.apache.org/jira/browse/SPARK-47923 Project: Spark

[jira] [Resolved] (ORC-1695) Upgrade gson to 2.10.1

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1695. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (ORC-1695) Upgrade gson to 2.10.1

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1695: -- Assignee: dzcxzl > Upgrade gson to 2.10.1 > -- > >

[jira] [Resolved] (SPARK-47825) Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47825. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46019 [https

[jira] [Assigned] (SPARK-47825) Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47825: - Assignee: BingKun Pan > Make `KinesisTestUtils` & `WriteInputFormatTestDataGe

[jira] [Updated] (SPARK-47901) Upgrade commons-text to 1.12.0

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47901: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade commons-t

[jira] [Updated] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47828: -- Affects Version/s: 3.4.3 3.4.1 3.4.0

[jira] [Updated] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47828: -- Fix Version/s: 3.4.4 > DataFrameWriterV2.overwrite fails with invalid p

[jira] [Updated] (SPARK-47915) Upgrade kubernetes-client to 6.12.1

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47915: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Dependency upgrade) > Upgr

[jira] [Resolved] (SPARK-47915) Upgrade kubernetes-client to 6.12.1

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47915. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46137 [https

[jira] [Assigned] (SPARK-47915) Upgrade kubernetes-client to 6.12.1

2024-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47915: - Assignee: Bjørn Jørgensen > Upgrade kubernetes-client to 6.1

[jira] [Resolved] (SPARK-47898) Port HIVE-12270: Add DBTokenStore support to HS2 delegation token

2024-04-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47898. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46115 [https

[ANNOUNCE] Apache Spark 3.4.3 released

2024-04-18 Thread Dongjoon Hyun
not have been possible without you. Dongjoon Hyun

[ANNOUNCE] Apache Spark 3.4.3 released

2024-04-18 Thread Dongjoon Hyun
not have been possible without you. Dongjoon Hyun

[jira] [Resolved] (SPARK-47889) Setup gradle as build tool for operator repository

2024-04-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47889. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 4 [https

[jira] [Assigned] (SPARK-47889) Setup gradle as build tool for operator repository

2024-04-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47889: - Assignee: Zhou JIANG > Setup gradle as build tool for operator reposit

[jira] [Updated] (SPARK-47889) Setup gradle as build tool for operator repository

2024-04-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47889: -- Fix Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Setup gra

[jira] [Assigned] (SPARK-47887) Remove unused import `spark/connect/common.proto` from `spark/connect/relations.proto`

2024-04-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47887: - Assignee: Yang Jie > Remove unused import `spark/connect/common.proto` from >

[jira] [Resolved] (SPARK-47887) Remove unused import `spark/connect/common.proto` from `spark/connect/relations.proto`

2024-04-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47887. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46106 [https

[jira] [Resolved] (SPARK-47893) Upgrade ASM to 9.7

2024-04-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47893. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46110 [https

[VOTE][RESULT] Release Spark 3.4.3 (RC2)

2024-04-18 Thread Dongjoon Hyun
The vote passes with 10 +1s (8 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Mridul Muralidharan * - Wenchen Fan * - Liang-Chi Hsieh * - Gengliang Wang * - Hyukjin Kwon * - Bo Yang - DB Tsai * - Kent Yao - Huaxin Gao * +0: None -1: None

<    1   2   3   4   5   6   7   8   9   10   >