[jira] [Updated] (SPARK-42929) make mapInPandas / mapInArrow support "is_barrier"

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-42929: --- Labels: pull-request-available (was: ) > make mapInPandas / mapInArrow support

[jira] [Comment Edited] (SPARK-47194) Upgrade log4j2 to 2.23.0

2024-02-27 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821543#comment-17821543 ] Yang Jie edited comment on SPARK-47194 at 2/28/24 7:19 AM: --- It seems that the

[jira] [Resolved] (SPARK-47194) Upgrade log4j2 to 2.23.0

2024-02-27 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-47194. -- Resolution: Won't Fix It seems that the `-Dlog4j2.debug` option may not be working in 2.23.0, perhaps

[jira] [Assigned] (SPARK-47199) Add prefix into TemporaryDirectory to avoid flakiness

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47199: - Assignee: Hyukjin Kwon > Add prefix into TemporaryDirectory to avoid flakiness >

[jira] [Resolved] (SPARK-47199) Add prefix into TemporaryDirectory to avoid flakiness

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47199. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45298

[jira] [Resolved] (SPARK-45200) Spark 3.4.0 always using default log4j profile

2024-02-27 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-45200. -- Resolution: Not A Problem > Spark 3.4.0 always using default log4j profile >

[jira] [Resolved] (SPARK-46766) ZSTD Buffer Pool Support For AVRO datasource

2024-02-27 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46766. -- Fix Version/s: 4.0.0 Resolution: Fixed resolved by https://github.com/apache/spark/pull/44792

[jira] [Assigned] (SPARK-46766) ZSTD Buffer Pool Support For AVRO datasource

2024-02-27 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-46766: Assignee: Kent Yao > ZSTD Buffer Pool Support For AVRO datasource >

[jira] [Updated] (SPARK-47197) Failed to connect HiveMetastore when using iceberg with HiveCatalog on spark-sql or spark-shell

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47197: --- Labels: pull-request-available (was: ) > Failed to connect HiveMetastore when using

[jira] [Assigned] (SPARK-47192) Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature)

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47192: Assignee: Serge Rielau > Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature) >

[jira] [Resolved] (SPARK-47192) Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature)

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47192. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45291

[jira] [Updated] (SPARK-47206) Add official image Dockerfile for Apache Spark 3.5.1

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47206: --- Labels: pull-request-available (was: ) > Add official image Dockerfile for Apache Spark

[jira] [Updated] (SPARK-47206) Add official image Dockerfile for Apache Spark 3.5.1

2024-02-27 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-47206: Summary: Add official image Dockerfile for Apache Spark 3.5.1 (was: [SPARK-45169] Add official

[jira] [Created] (SPARK-47206) [SPARK-45169] Add official image Dockerfile for Apache Spark 3.5.1

2024-02-27 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-47206: --- Summary: [SPARK-45169] Add official image Dockerfile for Apache Spark 3.5.1 Key: SPARK-47206 URL: https://issues.apache.org/jira/browse/SPARK-47206 Project: Spark

[jira] [Updated] (SPARK-47205) Upgrade docker-java to 3.3.5

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47205: --- Labels: pull-request-available (was: ) > Upgrade docker-java to 3.3.5 >

[jira] [Updated] (SPARK-47155) Fix incorrect error class in create_data_source.py

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47155: --- Labels: pull-request-available (was: ) > Fix incorrect error class in

[jira] [Created] (SPARK-47205) Upgrade docker-java to 3.3.5

2024-02-27 Thread Kent Yao (Jira)
Kent Yao created SPARK-47205: Summary: Upgrade docker-java to 3.3.5 Key: SPARK-47205 URL: https://issues.apache.org/jira/browse/SPARK-47205 Project: Spark Issue Type: Dependency upgrade

[jira] [Comment Edited] (SPARK-21918) HiveClient shouldn't share Hive object between different thread

2024-02-27 Thread Goutam Ghosh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821516#comment-17821516 ] Goutam Ghosh edited comment on SPARK-21918 at 2/28/24 5:58 AM: --- Can the

[jira] [Commented] (SPARK-21918) HiveClient shouldn't share Hive object between different thread

2024-02-27 Thread Goutam Ghosh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821516#comment-17821516 ] Goutam Ghosh commented on SPARK-21918: -- Can the path by [~angerszhuuu] be validated ? >

[jira] [Resolved] (SPARK-47191) avoid unnecessary relation lookup when uncaching table/view

2024-02-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47191. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45289

[jira] [Assigned] (SPARK-47191) avoid unnecessary relation lookup when uncaching table/view

2024-02-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47191: --- Assignee: Wenchen Fan > avoid unnecessary relation lookup when uncaching table/view >

[jira] [Resolved] (SPARK-47187) Fix hive compress output config does not work

2024-02-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-47187. --- Fix Version/s: 3.4.3 Resolution: Fixed Issue resolved by pull request 45286

[jira] [Assigned] (SPARK-47187) Fix hive compress output config does not work

2024-02-27 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-47187: - Assignee: XiDuo You > Fix hive compress output config does not work >

[jira] [Assigned] (SPARK-47202) AttributeError: module 'pandas' has no attribute 'Timstamp'

2024-02-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47202: Assignee: Arzav Jain > AttributeError: module 'pandas' has no attribute 'Timstamp' >

[jira] [Resolved] (SPARK-47202) AttributeError: module 'pandas' has no attribute 'Timstamp'

2024-02-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47202. -- Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-47144) Fix Spark Connect collation issue

2024-02-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47144. - Resolution: Fixed Issue resolved by pull request 45233

[jira] [Assigned] (SPARK-47144) Fix Spark Connect collation issue

2024-02-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47144: --- Assignee: Nikola Mandic > Fix Spark Connect collation issue >

[jira] [Resolved] (SPARK-47063) CAST long to timestamp has different behavior for codegen vs interpreted

2024-02-27 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-47063. -- Fix Version/s: 3.4.3 3.5.2 4.0.0 Resolution: Fixed Issue

[jira] [Assigned] (SPARK-47063) CAST long to timestamp has different behavior for codegen vs interpreted

2024-02-27 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-47063: Assignee: Pablo Langa Blanco > CAST long to timestamp has different behavior for codegen vs

[jira] [Resolved] (SPARK-47204) [CORE] Check whether enabled checksum before delete checksum file

2024-02-27 Thread Binjie Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binjie Yang resolved SPARK-47204. - Resolution: Not A Problem We will check whether the checksum file is exists or not before try

[jira] [Created] (SPARK-47204) [CORE] Check whether enabled checksum before delete checksum file

2024-02-27 Thread Binjie Yang (Jira)
Binjie Yang created SPARK-47204: --- Summary: [CORE] Check whether enabled checksum before delete checksum file Key: SPARK-47204 URL: https://issues.apache.org/jira/browse/SPARK-47204 Project: Spark

[jira] [Updated] (SPARK-47202) AttributeError: module 'pandas' has no attribute 'Timstamp'

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47202: --- Labels: pull-request-available (was: ) > AttributeError: module 'pandas' has no attribute

[jira] [Updated] (SPARK-47202) AttributeError: module 'pandas' has no attribute 'Timstamp'

2024-02-27 Thread Arzav Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arzav Jain updated SPARK-47202: --- Description: When using the pyspark.sql.types.TimestampType, if your value is a datetime.datetime

[jira] [Assigned] (SPARK-47201) sameSemantics check input types

2024-02-27 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47201: - Assignee: Ruifeng Zheng > sameSemantics check input types >

[jira] [Resolved] (SPARK-47201) sameSemantics check input types

2024-02-27 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47201. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45300

[jira] [Created] (SPARK-47202) AttributeError: module 'pandas' has no attribute 'Timstamp'

2024-02-27 Thread Arzav Jain (Jira)
Arzav Jain created SPARK-47202: -- Summary: AttributeError: module 'pandas' has no attribute 'Timstamp' Key: SPARK-47202 URL: https://issues.apache.org/jira/browse/SPARK-47202 Project: Spark

[jira] [Updated] (SPARK-47201) sameSemantics check input types

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47201: --- Labels: pull-request-available (was: ) > sameSemantics check input types >

[jira] [Created] (SPARK-47201) sameSemantics check input types

2024-02-27 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-47201: - Summary: sameSemantics check input types Key: SPARK-47201 URL: https://issues.apache.org/jira/browse/SPARK-47201 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-47186) Improve the debuggability for docker integration test

2024-02-27 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-47186. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45284

[jira] [Resolved] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47196. --- Fix Version/s: 3.4.3 Resolution: Fixed Issue resolved by pull request 45295

[jira] [Assigned] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47196: - Assignee: Dongjoon Hyun > Fix `core` module to succeed SBT tests >

[jira] [Updated] (SPARK-47200) Handle and classify errors from ForEachBatchSink user function

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47200: --- Labels: pull-request-available (was: ) > Handle and classify errors from ForEachBatchSink

[jira] [Created] (SPARK-47200) Handle and classify errors from ForEachBatchSink user function

2024-02-27 Thread B. Micheal Okutubo (Jira)
B. Micheal Okutubo created SPARK-47200: -- Summary: Handle and classify errors from ForEachBatchSink user function Key: SPARK-47200 URL: https://issues.apache.org/jira/browse/SPARK-47200 Project:

[jira] [Updated] (SPARK-47199) Add prefix into TemporaryDirectory to avoid flakiness

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47199: --- Labels: pull-request-available (was: ) > Add prefix into TemporaryDirectory to avoid

[jira] [Created] (SPARK-47199) Add prefix into TemporaryDirectory to avoid flakiness

2024-02-27 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-47199: Summary: Add prefix into TemporaryDirectory to avoid flakiness Key: SPARK-47199 URL: https://issues.apache.org/jira/browse/SPARK-47199 Project: Spark Issue

[jira] [Updated] (SPARK-47198) Is it possible to dynamically add backend service to ingress with Kubernetes?

2024-02-27 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-47198: -- Description: spark on k8s runs multiple spark apps at the same time. proxy/[sparkappid] path, forwarding to

[jira] [Created] (SPARK-47198) Is it possible to dynamically add backend service to ingress with Kubernetes?

2024-02-27 Thread melin (Jira)
melin created SPARK-47198: - Summary: Is it possible to dynamically add backend service to ingress with Kubernetes? Key: SPARK-47198 URL: https://issues.apache.org/jira/browse/SPARK-47198 Project: Spark

[jira] [Resolved] (SPARK-47114) In the spark driver pod. Failed to access the krb5 file

2024-02-27 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin resolved SPARK-47114. --- Resolution: Resolved > In the spark driver pod. Failed to access the krb5 file >

[jira] [Commented] (SPARK-47114) In the spark driver pod. Failed to access the krb5 file

2024-02-27 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821469#comment-17821469 ] melin commented on SPARK-47114: --- 默认jre17 不支持kerberos,换成jdk  > In the spark driver pod. Failed to access

[jira] [Updated] (SPARK-47197) Failed to connect HiveMetastore when using iceberg with HiveCatalog on spark-sql or spark-shell

2024-02-27 Thread YUBI LEE (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YUBI LEE updated SPARK-47197: - Description: I can't connect to kerberized HiveMetastore when using iceberg with HiveCatalog on

[jira] [Updated] (SPARK-47197) Failed to connect HiveMetastore when using iceberg with HiveCatalog on spark-sql or spark-shell

2024-02-27 Thread YUBI LEE (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YUBI LEE updated SPARK-47197: - Summary: Failed to connect HiveMetastore when using iceberg with HiveCatalog on spark-sql or

[jira] [Updated] (SPARK-47197) Failed to connect HiveMetastore when using iceberg with HiveCatalog by spark-sql or spark-shell

2024-02-27 Thread YUBI LEE (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YUBI LEE updated SPARK-47197: - Component/s: SQL > Failed to connect HiveMetastore when using iceberg with HiveCatalog by > spark-sql

[jira] [Commented] (SPARK-47172) Upgrade Transport block cipher mode to GCM

2024-02-27 Thread Steve Weis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821449#comment-17821449 ] Steve Weis commented on SPARK-47172: Hi [~mridul]. I'd like to get you input on this now that there

[jira] [Updated] (SPARK-47197) Failed to connect HiveMetastore when using iceberg with HiveCatalog by spark-sql or spark-shell

2024-02-27 Thread YUBI LEE (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YUBI LEE updated SPARK-47197: - Description: I can't connect to kerberized HiveMetastore when using iceberg with HiveCatalog by

[jira] [Updated] (SPARK-47197) Failed to connect HiveMetastore when using iceberg with HiveCatalog by spark-sql or spark-shell

2024-02-27 Thread YUBI LEE (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YUBI LEE updated SPARK-47197: - Description: I can't connect to kerberized HiveMetastore when using iceberg with HiveCatalog by

[jira] [Created] (SPARK-47197) Failed to connect HiveMetastore when using iceberg with HiveCatalog by spark-sql or spark-shell

2024-02-27 Thread YUBI LEE (Jira)
YUBI LEE created SPARK-47197: Summary: Failed to connect HiveMetastore when using iceberg with HiveCatalog by spark-sql or spark-shell Key: SPARK-47197 URL: https://issues.apache.org/jira/browse/SPARK-47197

[jira] [Commented] (SPARK-24815) Structured Streaming should support dynamic allocation

2024-02-27 Thread Mich Talebzadeh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821433#comment-17821433 ] Mich Talebzadeh commented on SPARK-24815: - some thoughts on this if I may   This enhancement

[jira] [Updated] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47196: -- Description: This happens at branch-3.4 only. branch-3.3/branch-3.5/master are okay.

[jira] [Updated] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47196: -- Description: This happens at branch-3.4 only. {code:java} $ build/sbt "core/testOnly

[jira] [Updated] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47196: -- Description: MAVEN {code} $ build/mvn dependency:tree -pl core | grep byte-buddy ... [INFO]

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-27 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-47177: - Description: AQE plan is expected to display final plan after execution. This is not true for cached

[jira] [Updated] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47196: -- Affects Version/s: 3.4.1 3.4.0 > Fix `core` module to succeed SBT

[jira] [Updated] (SPARK-43157) TreeNode tags can become corrupted and hang driver when the dataset is cached

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43157: --- Labels: pull-request-available (was: ) > TreeNode tags can become corrupted and hang

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-27 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-47177: - Description: AQE plan is expected to display final plan after execution. This is not true for cached

[jira] [Updated] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

2024-02-27 Thread Ziqi Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziqi Liu updated SPARK-47177: - Description: AQE plan is expected to display final plan after execution. This is not true for cached

[jira] [Updated] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47196: --- Labels: pull-request-available (was: ) > Fix `core` module to succeed SBT tests >

[jira] [Created] (SPARK-47196) Fix `core` module to succeed SBT tests

2024-02-27 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-47196: - Summary: Fix `core` module to succeed SBT tests Key: SPARK-47196 URL: https://issues.apache.org/jira/browse/SPARK-47196 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821393#comment-17821393 ] Bruce Robbins edited comment on SPARK-47193 at 2/27/24 8:48 PM: Running

[jira] [Updated] (SPARK-47063) CAST long to timestamp has different behavior for codegen vs interpreted

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47063: --- Labels: pull-request-available (was: ) > CAST long to timestamp has different behavior for

[jira] [Resolved] (SPARK-43256) Assign a name to the error class _LEGACY_ERROR_TEMP_2021

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43256. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45198

[jira] [Assigned] (SPARK-43256) Assign a name to the error class _LEGACY_ERROR_TEMP_2021

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-43256: Assignee: A G > Assign a name to the error class _LEGACY_ERROR_TEMP_2021 >

[jira] [Commented] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821393#comment-17821393 ] Bruce Robbins commented on SPARK-47193: --- Running this in Spark 3.5.0 in local mode on my laptop, I

[jira] [Commented] (SPARK-44389) ExecutorDeadException when using decommissioning without external shuffle service

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821373#comment-17821373 ] Dongjoon Hyun commented on SPARK-44389: --- I guess this was fixed by reverting SPARK-43043 via

[jira] [Comment Edited] (SPARK-44478) Executor decommission causes stage failure

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821370#comment-17821370 ] Dongjoon Hyun edited comment on SPARK-44478 at 2/27/24 6:51 PM: Do you

[jira] [Commented] (SPARK-44478) Executor decommission causes stage failure

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821370#comment-17821370 ] Dongjoon Hyun commented on SPARK-44478: --- Do you still see this issue with Apache Spark 3.4.2,

[jira] [Commented] (SPARK-47194) Upgrade log4j2 to 2.23.0

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821367#comment-17821367 ] Dongjoon Hyun commented on SPARK-47194: --- Hi, [~LuciferYang]. Thank you for adding here.

[jira] [Comment Edited] (SPARK-47033) EXECUTE IMMEDIATE USING does not recognize session variable names

2024-02-27 Thread A G (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820680#comment-17820680 ] A G edited comment on SPARK-47033 at 2/27/24 6:08 PM: -- I want to work on this! PR:

[jira] [Updated] (SPARK-47033) EXECUTE IMMEDIATE USING does not recognize session variable names

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47033: --- Labels: pull-request-available (was: ) > EXECUTE IMMEDIATE USING does not recognize

[jira] [Updated] (SPARK-47194) Upgrade log4j2 to 2.23.0

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47194: --- Labels: pull-request-available (was: ) > Upgrade log4j2 to 2.23.0 >

[jira] [Created] (SPARK-47194) Upgrade log4j2 to 2.23.0

2024-02-27 Thread Yang Jie (Jira)
Yang Jie created SPARK-47194: Summary: Upgrade log4j2 to 2.23.0 Key: SPARK-47194 URL: https://issues.apache.org/jira/browse/SPARK-47194 Project: Spark Issue Type: Sub-task Components:

[jira] [Updated] (SPARK-47192) Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature)

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47192: --- Labels: pull-request-available (was: ) > Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive

[jira] [Updated] (SPARK-46834) Aggregate support for strings with collation

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46834: --- Labels: pull-request-available (was: ) > Aggregate support for strings with collation >

[jira] [Updated] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Ivan Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Bova updated SPARK-47193: -- Attachment: device.csv deviceClass.csv deviceType.csv

[jira] [Created] (SPARK-47193) Converting dataframe to rdd results in data loss

2024-02-27 Thread Ivan Bova (Jira)
Ivan Bova created SPARK-47193: - Summary: Converting dataframe to rdd results in data loss Key: SPARK-47193 URL: https://issues.apache.org/jira/browse/SPARK-47193 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-47192) Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature)

2024-02-27 Thread Serge Rielau (Jira)
Serge Rielau created SPARK-47192: Summary: Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature) Key: SPARK-47192 URL: https://issues.apache.org/jira/browse/SPARK-47192 Project: Spark

[jira] [Assigned] (SPARK-47185) Increase timeout between actions in KafkaContinuousTest

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47185: - Assignee: Hyukjin Kwon > Increase timeout between actions in KafkaContinuousTest >

[jira] [Resolved] (SPARK-47185) Increase timeout between actions in KafkaContinuousTest

2024-02-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47185. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45283

[jira] [Updated] (SPARK-47191) avoid unnecessary relation lookup when uncaching table/view

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47191: --- Labels: pull-request-available (was: ) > avoid unnecessary relation lookup when uncaching

[jira] [Created] (SPARK-47191) avoid unnecessary relation lookup when uncaching table/view

2024-02-27 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-47191: --- Summary: avoid unnecessary relation lookup when uncaching table/view Key: SPARK-47191 URL: https://issues.apache.org/jira/browse/SPARK-47191 Project: Spark

[jira] [Commented] (SPARK-47190) Add support for checkpointing to Spark Connect

2024-02-27 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821286#comment-17821286 ] Nicholas Chammas commented on SPARK-47190: -- [~gurwls223] - Is there some design reason we do

[jira] [Created] (SPARK-47190) Add support for checkpointing to Spark Connect

2024-02-27 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-47190: Summary: Add support for checkpointing to Spark Connect Key: SPARK-47190 URL: https://issues.apache.org/jira/browse/SPARK-47190 Project: Spark Issue

[jira] [Assigned] (SPARK-47189) Tweak column error names and text

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47189: Assignee: Nicholas Chammas > Tweak column error names and text >

[jira] [Resolved] (SPARK-47189) Tweak column error names and text

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47189. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45276

[jira] [Commented] (SPARK-45527) Task fraction resource request is not expected

2024-02-27 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821279#comment-17821279 ] Thomas Graves commented on SPARK-45527: --- Note that this is related to SPARK-39853 which was

[jira] [Commented] (SPARK-47063) CAST long to timestamp has different behavior for codegen vs interpreted

2024-02-27 Thread Pablo Langa Blanco (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821273#comment-17821273 ] Pablo Langa Blanco commented on SPARK-47063: Ok, personally I would prefer the truncation

[jira] [Updated] (SPARK-47189) Tweak column error names and text

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47189: --- Labels: pull-request-available (was: ) > Tweak column error names and text >

[jira] [Created] (SPARK-47189) Tweak column error names and text

2024-02-27 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-47189: Summary: Tweak column error names and text Key: SPARK-47189 URL: https://issues.apache.org/jira/browse/SPARK-47189 Project: Spark Issue Type:

[jira] [Updated] (SPARK-47145) Provide table identifier to scan node when DS v2 strategy is applied

2024-02-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47145: --- Labels: pull-request-available (was: ) > Provide table identifier to scan node when DS v2

[jira] [Resolved] (SPARK-47179) Improve error message from SparkThrowableSuite for better debuggability

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47179. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45273

[jira] [Assigned] (SPARK-47179) Improve error message from SparkThrowableSuite for better debuggability

2024-02-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-47179: Assignee: Haejoon Lee > Improve error message from SparkThrowableSuite for better debuggability

  1   2   >