[jira] [Updated] (SPARK-48068) Fix `mypy` failure in Python 3.10+

2024-05-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48068: -- Description: {code} $ python3 --version Python 3.10.13 $ dev/lint-python --mypy starting

[jira] [Updated] (SPARK-48068) Fix `mypy` failure in Python 3.10+

2024-05-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48068: -- Summary: Fix `mypy` failure in Python 3.10+ (was: Fix `mypy` failure in Python 3.10) >

[jira] [Updated] (SPARK-48068) Fix `mypy` failure in Python 3.10

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48068: -- Description: {code} $ python3 --version Python 3.10.13 $ dev/lint-python --mypy starting

[jira] [Created] (SPARK-48068) Fix `mypy` failure in Python 3.10

2024-04-30 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48068: - Summary: Fix `mypy` failure in Python 3.10 Key: SPARK-48068 URL: https://issues.apache.org/jira/browse/SPARK-48068 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-48028) Regenerate benchmark results after turning ANSI on

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48028. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46266 [https

[jira] [Resolved] (SPARK-48063) Enable `spark.stage.ignoreDecommissionFetchFailure` by default

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48063. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46308 [https

[jira] [Assigned] (SPARK-48063) Enable `spark.stage.ignoreDecommissionFetchFailure` by default

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48063: - Assignee: Dongjoon Hyun > Enable `spark.stage.ignoreDecommissionFetchFail

[jira] [Created] (SPARK-48063) Enable `spark.stage.ignoreDecommissionFetchFailure` by default

2024-04-30 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48063: - Summary: Enable `spark.stage.ignoreDecommissionFetchFailure` by default Key: SPARK-48063 URL: https://issues.apache.org/jira/browse/SPARK-48063 Project: Spark

[jira] [Updated] (SPARK-48060) Fix StreamingQueryHashPartitionVerifySuite to update golden files correctly

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48060: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Test) >

[jira] [Resolved] (SPARK-48060) Fix StreamingQueryHashPartitionVerifySuite to update golden files correctly

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48060. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46304 [https

[jira] [Resolved] (SPARK-48057) Enable `GroupedApplyInPandasTests. test_grouped_with_empty_partition`

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48057. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46299 [https

[jira] [Resolved] (SPARK-48061) Parameterize max limits of `spark.sql.test.randomDataGenerator`

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48061. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46305 [https

[jira] [Assigned] (SPARK-48061) Parameterize max limits of `spark.sql.test.randomDataGenerator`

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48061: - Assignee: Dongjoon Hyun > Parameterize max lim

[jira] [Updated] (SPARK-48061) Parameterize max limits of `spark.sql.test.randomDataGenerator`

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48061: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Test) > Parameterize max lim

[jira] [Created] (SPARK-48061) Parameterize max limits of `spark.sql.test.randomDataGenerator`

2024-04-30 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48061: - Summary: Parameterize max limits of `spark.sql.test.randomDataGenerator` Key: SPARK-48061 URL: https://issues.apache.org/jira/browse/SPARK-48061 Project: Spark

[jira] [Created] (SPARK-48060) Fix StreamingQueryHashPartitionVerifySuite to update golden files correctly

2024-04-30 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48060: - Summary: Fix StreamingQueryHashPartitionVerifySuite to update golden files correctly Key: SPARK-48060 URL: https://issues.apache.org/jira/browse/SPARK-48060

[jira] [Resolved] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default

2024-04-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46122. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46207 [https

[VOTE][RESULT] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-30 Thread Dongjoon Hyun
The vote passes with 11 +1s (6 binding +1s) and one -1. Thanks to all who helped with the vote! (* = binding) +1: - Dongjoon Hyun * - Gengliang Wang * - Liang-Chi Hsieh * - Holden Karau * - Zhou Jiang - Cheng Pan - Hyukjin Kwon * - DB Tsai * - Ye Xianjin - XiDuo You - Nimrod Ofek +0: None -1

[jira] [Commented] (SPARK-48016) Fix a bug in try_divide function when with decimals

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842200#comment-17842200 ] Dongjoon Hyun commented on SPARK-48016: --- Thank you so much! > Fix a bug in try_divide funct

[jira] [Commented] (SPARK-48016) Fix a bug in try_divide function when with decimals

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842191#comment-17842191 ] Dongjoon Hyun commented on SPARK-48016: --- Hi, [~Gengliang.Wang] . - I updated the JIRA title

[jira] [Updated] (SPARK-48016) Fix a bug in try_divide function when with decimals

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48016: -- Summary: Fix a bug in try_divide function when with decimals (was: Binary Arithmetic

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-29 Thread Dongjoon Hyun
sand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Mon, 29 Apr 2024 at 17:32, Dongjoon Hyun wrote: > > > It's a surprise to me to see t

[jira] [Resolved] (SPARK-48042) Don't use a copy of timestamp formatter with a new override zone for each value

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48042. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46282 [https

[jira] [Resolved] (SPARK-48044) Cache `DataFrame.isStreaming`

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48044. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46281 [https

[jira] [Resolved] (SPARK-48046) Remove `clock` parameter from `DriverServiceFeatureStep`

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48046. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46284 [https

[jira] [Assigned] (SPARK-48046) Remove `clock` parameter from `DriverServiceFeatureStep`

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48046: - Assignee: Dongjoon Hyun > Remove `clock` parameter from `DriverServiceFeatureS

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-29 Thread Dongjoon Hyun
t; 1) Hive provides a more mature and widely adopted catalog >>>>>>>>>>> solution that integrates well with other components in the Hadoop >>>>>>>>>>> ecosystem, such as HDFS, HBase, and YARN. IIf you are Hadoop >&g

[jira] [Created] (SPARK-48046) Remove `clock` parameter from `DriverServiceFeatureStep`

2024-04-29 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48046: - Summary: Remove `clock` parameter from `DriverServiceFeatureStep` Key: SPARK-48046 URL: https://issues.apache.org/jira/browse/SPARK-48046 Project: Spark

[jira] [Assigned] (SPARK-48038) Promote driverServiceName to KubernetesDriverConf

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48038: - Assignee: Cheng Pan > Promote driverServiceName to KubernetesDriverC

[jira] [Resolved] (SPARK-48038) Promote driverServiceName to KubernetesDriverConf

2024-04-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48038. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46276 [https

[jira] [Resolved] (SPARK-48036) Update `sql-ref-ansi-compliance.md` and `sql-ref-identifier.md`

2024-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48036. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46271 [https

[jira] [Resolved] (SPARK-48029) Update the packages name removed in building the spark docker image

2024-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48029. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46258 [https

[jira] [Assigned] (SPARK-48029) Update the packages name removed in building the spark docker image

2024-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48029: - Assignee: BingKun Pan > Update the packages name removed in building the spark doc

[jira] [Created] (SPARK-48036) Update `sql-ref-ansi-compliance.md` and `sql-ref-identifier.md`

2024-04-28 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48036: - Summary: Update `sql-ref-ansi-compliance.md` and `sql-ref-identifier.md` Key: SPARK-48036 URL: https://issues.apache.org/jira/browse/SPARK-48036 Project: Spark

[jira] [Resolved] (SPARK-48032) Upgrade `commons-codec` to 1.17.0

2024-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48032. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46268 [https

[jira] [Updated] (SPARK-47730) Support APP_ID and EXECUTOR_ID placeholder in labels

2024-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47730: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Support APP

[jira] [Resolved] (SPARK-47730) Support APP_ID and EXECUTOR_ID placeholder in labels

2024-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47730. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46149 [https

[jira] [Assigned] (SPARK-47730) Support APP_ID and EXECUTOR_ID placeholder in labels

2024-04-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47730: - Assignee: Xi Chen > Support APP_ID and EXECUTOR_ID placeholder in lab

[jira] [Assigned] (SPARK-48021) Add `--add-modules=jdk.incubator.vector` to `JavaModuleOptions`

2024-04-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48021: - Assignee: BingKun Pan > Add `--add-modules=jdk.incubator.vector` to `JavaModuleOpti

[jira] [Resolved] (SPARK-48021) Add `--add-modules=jdk.incubator.vector` to `JavaModuleOptions`

2024-04-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48021. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46246 [https

[jira] [Resolved] (SPARK-47408) Fix mathExpressions that use StringType

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47408. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46227 [https

[jira] [Updated] (SPARK-47943) Add Operator CI Task for Java Build and Test

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47943: -- Fix Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Add Opera

[jira] [Assigned] (SPARK-48015) Update `build.gradle` to fix deprecation warnings

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48015: - Assignee: Dongjoon Hyun > Update `build.gradle` to fix deprecation warni

[jira] [Resolved] (SPARK-48015) Update `build.gradle` to fix deprecation warnings

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48015. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 9 [https

[jira] [Updated] (SPARK-47929) Setup Static Analysis for Operator

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47929: -- Fix Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Setup Sta

[jira] [Updated] (SPARK-47950) Add Java API Module for Spark Operator

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47950: -- Fix Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Add Java

[jira] [Updated] (SPARK-48015) Update `build.gradle` to fix deprecation warnings

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48015: -- Fix Version/s: kubernetes-operator-0.1.0 (was: 4.0.0) > Upd

[jira] [Created] (SPARK-48015) Update `build.gradle` to fix deprecation warnings

2024-04-26 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-48015: - Summary: Update `build.gradle` to fix deprecation warnings Key: SPARK-48015 URL: https://issues.apache.org/jira/browse/SPARK-48015 Project: Spark Issue

[jira] [Assigned] (SPARK-47950) Add Java API Module for Spark Operator

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47950: - Assignee: Zhou JIANG > Add Java API Module for Spark Opera

[jira] [Resolved] (SPARK-47950) Add Java API Module for Spark Operator

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47950. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 8 [https

[jira] [Resolved] (SPARK-48010) Avoid repeated calls to conf.resolver in resolveExpression

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48010. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46248 [https

[jira] [Assigned] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46122: - Assignee: Dongjoon Hyun > Set `spark.sql.legacy.createHiveTableByDefault` to `fa

[jira] [Resolved] (SPARK-48005) Enable `DefaultIndexParityTests. test_index_distributed_sequence_cleanup`

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48005. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46242 [https

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-26 Thread Dongjoon Hyun
I'll start with my +1. Dongjoon. On 2024/04/26 16:45:51 Dongjoon Hyun wrote: > Please vote on SPARK-46122 to set spark.sql.legacy.createHiveTableByDefault > to `false` by default. The technical scope is defined in the following PR. > > - DISCUSSION: > https://lists.ap

[VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-26 Thread Dongjoon Hyun
Please vote on SPARK-46122 to set spark.sql.legacy.createHiveTableByDefault to `false` by default. The technical scope is defined in the following PR. - DISCUSSION: https://lists.apache.org/thread/ylk96fg4lvn6klxhj6t6yh42lyqb8wmd - JIRA: https://issues.apache.org/jira/browse/SPARK-46122 - PR:

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-26 Thread Dongjoon Hyun
;>>> complex >>>>>>>>>> SQL queries or existing SQL-based workflows, using Hive may be >>>>>>>>>> advantageous. >>>>>>>>>> 3) If you are looking for performance, spark's native catalog >>>>

[jira] [Assigned] (ORC-1705) Upgrade `zstd-jni` to 1.5.6-3

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1705: -- Assignee: dzcxzl > Upgrade `zstd-jni` to 1.5.

[jira] [Resolved] (ORC-1705) Upgrade `zstd-jni` to 1.5.6-3

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1705. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-48007) MsSQLServer: upgrade mssql.jdbc.version to 12.6.1.jre11

2024-04-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48007. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46244 [https

[jira] [Resolved] (SPARK-47991) Arrange the test cases for window frames and window functions.

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47991. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46226 [https

[jira] [Commented] (SPARK-22231) Support of map, filter, withField, dropFields in nested list of structures

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840934#comment-17840934 ] Dongjoon Hyun commented on SPARK-22231: --- I removed the outdated target version from this issue

[jira] [Updated] (SPARK-22231) Support of map, filter, withField, dropFields in nested list of structures

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22231: -- Target Version/s: (was: 3.2.0) > Support of map, filter, withField, dropFields in nes

[jira] [Updated] (SPARK-24941) Add RDDBarrier.coalesce() function

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24941: -- Target Version/s: (was: 3.2.0) > Add RDDBarrier.coalesce() funct

[jira] [Updated] (SPARK-25383) Image data source supports sample pushdown

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25383: -- Target Version/s: (was: 3.2.0) > Image data source supports sample pushd

[jira] [Updated] (SPARK-25752) Add trait to easily whitelist logical operators that produce named output from CleanupAliases

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25752: -- Target Version/s: (was: 3.2.0) > Add trait to easily whitelist logical operat

[jira] [Commented] (SPARK-28629) Capture the missing rules in HiveSessionStateBuilder

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840928#comment-17840928 ] Dongjoon Hyun commented on SPARK-28629: --- I removed the outdated target version from this issue

[jira] [Commented] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840930#comment-17840930 ] Dongjoon Hyun commented on SPARK-27780: --- I removed the outdated target version from this issue

[jira] [Updated] (SPARK-28629) Capture the missing rules in HiveSessionStateBuilder

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28629: -- Target Version/s: (was: 3.2.0) > Capture the missing rules in HiveSessionStateBuil

[jira] [Updated] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27780: -- Target Version/s: (was: 3.2.0) > Shuffle server & client should be versioned to

[jira] [Commented] (SPARK-30324) Simplify API for JSON access in DataFrames/SQL

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840927#comment-17840927 ] Dongjoon Hyun commented on SPARK-30324: --- I removed the outdated target version from this issue

[jira] [Updated] (SPARK-30324) Simplify API for JSON access in DataFrames/SQL

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30324: -- Target Version/s: (was: 3.2.0) > Simplify API for JSON access in DataFrames/

[jira] [Updated] (SPARK-30334) Add metadata around semi-structured columns to Spark

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30334: -- Target Version/s: (was: 3.2.0) > Add metadata around semi-structured columns to Sp

[jira] [Commented] (SPARK-30334) Add metadata around semi-structured columns to Spark

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840926#comment-17840926 ] Dongjoon Hyun commented on SPARK-30334: --- I removed the outdated target version from this issue

[jira] [Commented] (SPARK-24942) Improve cluster resource management with jobs containing barrier stage

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840913#comment-17840913 ] Dongjoon Hyun commented on SPARK-24942: --- I removed the outdated target version, `3.2.0`, from

[jira] [Updated] (SPARK-24942) Improve cluster resource management with jobs containing barrier stage

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24942: -- Target Version/s: (was: 3.2.0) > Improve cluster resource management with jobs contain

[jira] [Commented] (SPARK-44111) Prepare Apache Spark 4.0.0

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840853#comment-17840853 ] Dongjoon Hyun commented on SPARK-44111: --- Yes, we will provide `4.0.0-preview` in advance

[FYI] SPARK-47993: Drop Python 3.8

2024-04-25 Thread Dongjoon Hyun
FYI, there is a proposal to drop Python 3.8 because its EOL is October 2024. https://github.com/apache/spark/pull/46228 [SPARK-47993][PYTHON] Drop Python 3.8 Since it's still alive and there will be an overlap between the lifecycle of Python 3.8 and Apache Spark 4.0.0, please give us your

[jira] [Resolved] (SPARK-47987) Enable `ArrowParityTests.test_createDataFrame_empty_partition`

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47987. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46220 [https

[jira] [Resolved] (SPARK-47990) Upgrade `zstd-jni` to 1.5.6-3

2024-04-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47990. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46225 [https

[jira] [Commented] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840644#comment-17840644 ] Dongjoon Hyun commented on SPARK-46122: --- I sent the discussion thread for this issue. - [https

[jira] [Resolved] (ORC-1704) Migration to Scala 2.13 of Apache Spark 3.5.1 at SparkBenchmark

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1704. Fix Version/s: 2.0.1 2.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (ORC-1704) Migration to Scala 2.13 of Apache Spark 3.5.1 at SparkBenchmark

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1704: -- Assignee: dzcxzl > Migration to Scala 2.13 of Apache Spark 3.5.1 at SparkBenchm

[DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

2024-04-24 Thread Dongjoon Hyun
Hi, All. It's great to see community activities to polish 4.0.0 more and more. Thank you all. I'd like to bring SPARK-46122 (another SQL topic) to you from the subtasks of SPARK-4 (Prepare Apache Spark 4.0.0), - https://issues.apache.org/jira/browse/SPARK-46122 Set

[jira] [Updated] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46122: -- Summary: Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default (was: Set

[jira] [Updated] (SPARK-46122) Set `spark.sql.legacy.createHiveTableByDefault` to false by default

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46122: -- Summary: Set `spark.sql.legacy.createHiveTableByDefault` to false by default (was: Disable

[jira] [Resolved] (SPARK-47979) Use Hive tables explicitly for Hive table capability tests

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47979. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46211 [https

[jira] [Created] (SPARK-47979) Use Hive table explicitly for Hive table capability tests

2024-04-24 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47979: - Summary: Use Hive table explicitly for Hive table capability tests Key: SPARK-47979 URL: https://issues.apache.org/jira/browse/SPARK-47979 Project: Spark

[jira] [Updated] (SPARK-47979) Use Hive tables explicitly for Hive table capability tests

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47979: -- Summary: Use Hive tables explicitly for Hive table capability tests (was: Use Hive table

[jira] [Assigned] (SPARK-45265) Support Hive 4.0 metastore

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45265: - Assignee: (was: Attila Zsolt Piros) > Support Hive 4.0 metast

[jira] [Updated] (SPARK-44677) Drop legacy Hive-based ORC file format

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44677: -- Parent: (was: SPARK-44111) Issue Type: Task (was: Sub-task) > Drop legacy H

[jira] [Commented] (SPARK-47499) Reuse `test_help_command` in Connect

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840514#comment-17840514 ] Dongjoon Hyun commented on SPARK-47499: --- Thank you for collecting this to the umbrella Jira

[jira] [Assigned] (SPARK-47633) Cache miss for queries using JOIN LATERAL with join condition

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47633: - Assignee: Bruce Robbins > Cache miss for queries using JOIN LATERAL with j

[jira] [Resolved] (SPARK-47633) Cache miss for queries using JOIN LATERAL with join condition

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47633. --- Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46190 [https

Re: [VOTE] Apache Hive 2.3.10 Release Candidate 0

2024-04-24 Thread Dongjoon Hyun
Hi, Chao. The vote email seems to have a wrong repository link, 1106. IIUC, 1128 is the valid one, isn't it? Dongjoon. On 2024/04/20 20:01:53 Chao Sun wrote: > Apache Hive 2.3.10 Release Candidate 0 is available here: > https://people.apache.org/~sunchao/apache-hive-2.3.10-rc-0/ > Maven

[jira] [Resolved] (ORC-1703) Bump `maven-jar-plugin` to 3.4.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1703. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1905 [https

[jira] [Assigned] (ORC-1703) Bump `maven-jar-plugin` to 3.4.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1703: -- Assignee: Dongjoon Hyun > Bump `maven-jar-plugin` to 3.

[jira] [Created] (ORC-1703) Bump `maven-jar-plugin` to 3.4.1

2024-04-24 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created ORC-1703: -- Summary: Bump `maven-jar-plugin` to 3.4.1 Key: ORC-1703 URL: https://issues.apache.org/jira/browse/ORC-1703 Project: ORC Issue Type: Bug Components

[jira] [Resolved] (ORC-1702) Bump `bcpkix-jdk18on` to 1.78.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved ORC-1702. Fix Version/s: 2.1.0 Resolution: Fixed Issue resolved by pull request 1904 [https

[jira] [Assigned] (ORC-1702) Bump `bcpkix-jdk18on` to 1.78.1

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1702: -- Assignee: Dongjoon Hyun > Bump `bcpkix-jdk18on` to 1.7

[jira] [Assigned] (ORC-1701) Bump `threeten-extra` to 1.8.0

2024-04-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/ORC-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned ORC-1701: -- Assignee: Dongjoon Hyun > Bump `threeten-extra` to 1.

<    1   2   3   4   5   6   7   8   9   10   >