[jira] [Updated] (SPARK-46263) Clean up unnecessary `SeqOps.view` and `ArrayOps.view` conversions.

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46263: --- Labels: pull-request-available (was: ) > Clean up unnecessary `SeqOps.view` and

[jira] [Created] (SPARK-46263) Clean up unnecessary `SeqOps.view` and `ArrayOps.view` conversions.

2023-12-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-46263: Summary: Clean up unnecessary `SeqOps.view` and `ArrayOps.view` conversions. Key: SPARK-46263 URL: https://issues.apache.org/jira/browse/SPARK-46263 Project: Spark

[jira] [Updated] (SPARK-46262) Support `np.left_shift` for Pandas-on-Spark object.

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46262: --- Labels: pull-request-available (was: ) > Support `np.left_shift` for Pandas-on-Spark

[jira] [Created] (SPARK-46262) Support `np.left_shift` for Pandas-on-Spark object.

2023-12-04 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46262: --- Summary: Support `np.left_shift` for Pandas-on-Spark object. Key: SPARK-46262 URL: https://issues.apache.org/jira/browse/SPARK-46262 Project: Spark Issue

[jira] [Resolved] (SPARK-46249) Fix state store metrics access after commit

2023-12-04 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-46249. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44165

[jira] [Assigned] (SPARK-46249) Fix state store metrics access after commit

2023-12-04 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-46249: Assignee: Anish Shrigondekar > Fix state store metrics access after commit >

[jira] [Updated] (SPARK-46260) DataFrame.withColumnsRenamed should respect the dict ordering

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46260: --- Labels: pull-request-available (was: ) > DataFrame.withColumnsRenamed should respect the

[jira] [Updated] (SPARK-46257) Upgrade Derby to 10.16.1.1

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46257: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Upgrade Derby to

[jira] [Created] (SPARK-46261) Python Client DataFrame.withColumnsRenamed should respect the dict ordering

2023-12-04 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46261: - Summary: Python Client DataFrame.withColumnsRenamed should respect the dict ordering Key: SPARK-46261 URL: https://issues.apache.org/jira/browse/SPARK-46261

[jira] [Created] (SPARK-46260) DataFrame.withColumnsRenamed should respect the dict ordering

2023-12-04 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46260: - Summary: DataFrame.withColumnsRenamed should respect the dict ordering Key: SPARK-46260 URL: https://issues.apache.org/jira/browse/SPARK-46260 Project: Spark

[jira] [Assigned] (SPARK-46258) Add RocksDBPersistenceEngine

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46258: - Assignee: Dongjoon Hyun > Add RocksDBPersistenceEngine >

[jira] [Updated] (SPARK-46001) Spark UI Test Improvements

2023-12-04 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46001: - Shepherd: Dongjoon Hyun > Spark UI Test Improvements > -- > >

[jira] [Updated] (SPARK-46259) Add appropriate link for error class usage documentation.

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46259: --- Labels: pull-request-available (was: ) > Add appropriate link for error class usage

[jira] [Updated] (SPARK-46257) Upgrade Derby to 10.16.1.1

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46257: --- Labels: pull-request-available (was: ) > Upgrade Derby to 10.16.1.1 >

[jira] [Created] (SPARK-46259) Add appropriate link for error class usage documentation.

2023-12-04 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46259: --- Summary: Add appropriate link for error class usage documentation. Key: SPARK-46259 URL: https://issues.apache.org/jira/browse/SPARK-46259 Project: Spark

[jira] [Updated] (SPARK-46258) Add RocksDBPersistenceEngine

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46258: --- Labels: pull-request-available (was: ) > Add RocksDBPersistenceEngine >

[jira] [Created] (SPARK-46258) Add RocksDBPersistenceEngine

2023-12-04 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46258: - Summary: Add RocksDBPersistenceEngine Key: SPARK-46258 URL: https://issues.apache.org/jira/browse/SPARK-46258 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-46257) Upgrade Derby to 10.16.1.1

2023-12-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-46257: - Summary: Upgrade Derby to 10.16.1.1 (was: Upgrade Derby to 10.17.1.0) > Upgrade Derby to 10.16.1.1 >

[jira] [Updated] (SPARK-46256) Parallel Compression Support for ZSTD

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46256: --- Labels: pull-request-available (was: ) > Parallel Compression Support for ZSTD >

[jira] [Created] (SPARK-46257) Upgrade Derby to 10.17.1.0

2023-12-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-46257: Summary: Upgrade Derby to 10.17.1.0 Key: SPARK-46257 URL: https://issues.apache.org/jira/browse/SPARK-46257 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-46256) Parallel Compression Support for ZSTD

2023-12-04 Thread Kent Yao (Jira)
Kent Yao created SPARK-46256: Summary: Parallel Compression Support for ZSTD Key: SPARK-46256 URL: https://issues.apache.org/jira/browse/SPARK-46256 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-46254) Remove stale Python 3.8/3.7 version checkings

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46254. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44169

[jira] [Created] (SPARK-46255) Support complex type -> string conversion

2023-12-04 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46255: - Summary: Support complex type -> string conversion Key: SPARK-46255 URL: https://issues.apache.org/jira/browse/SPARK-46255 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46253) Plan Python data source read using mapInArrow

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46253: --- Labels: pull-request-available (was: ) > Plan Python data source read using mapInArrow >

[jira] [Assigned] (SPARK-46043) Support create table using DSv2 sources

2023-12-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-46043: --- Assignee: Allison Wang > Support create table using DSv2 sources >

[jira] [Resolved] (SPARK-46043) Support create table using DSv2 sources

2023-12-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-46043. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43949

[jira] [Reopened] (SPARK-46213) Add PySparkImportError for error framework

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-46213: -- Assignee: (was: Haejoon Lee) Reverted at

[jira] [Updated] (SPARK-46213) Add PySparkImportError for error framework

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46213: - Fix Version/s: (was: 4.0.0) > Add PySparkImportError for error framework >

[jira] [Resolved] (SPARK-46009) Merge the parse rule of PercentileCont and PercentileDisc into functionCall

2023-12-04 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng resolved SPARK-46009. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43910

[jira] [Updated] (SPARK-46254) Remove stale Python 3.8/3.7 version checkings

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46254: --- Labels: pull-request-available (was: ) > Remove stale Python 3.8/3.7 version checkings >

[jira] [Updated] (SPARK-46254) Remove stale Python 3.8/3.7 version checkings

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46254: - Issue Type: Improvement (was: New Feature) > Remove stale Python 3.8/3.7 version checkings >

[jira] [Updated] (SPARK-46254) Remove stale Python 3.8/3.7 version checkings

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46254: - Description: See PR linked. We dropped Python 3.7 and lowest is Python 3.8 so we can remove all

[jira] [Created] (SPARK-46254) Remove stale Python 3.8/3.7 version checkings

2023-12-04 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-46254: Summary: Remove stale Python 3.8/3.7 version checkings Key: SPARK-46254 URL: https://issues.apache.org/jira/browse/SPARK-46254 Project: Spark Issue Type:

[jira] [Created] (SPARK-46253) Plan Python data source read using mapInArrow

2023-12-04 Thread Allison Wang (Jira)
Allison Wang created SPARK-46253: Summary: Plan Python data source read using mapInArrow Key: SPARK-46253 URL: https://issues.apache.org/jira/browse/SPARK-46253 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-46040) Update API for 'analyze' partitioning/ordering columns to support general expressions

2023-12-04 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-46040. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43946

[jira] [Assigned] (SPARK-46233) Migrate all remaining ArrtibuteError into PySpark error framework

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46233: - Assignee: Haejoon Lee > Migrate all remaining ArrtibuteError into PySpark error

[jira] [Resolved] (SPARK-46233) Migrate all remaining ArrtibuteError into PySpark error framework

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46233. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44150

[jira] [Assigned] (SPARK-46250) Deflake test_parity_listener

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46250: Assignee: Wei Liu > Deflake test_parity_listener > > >

[jira] [Resolved] (SPARK-46250) Deflake test_parity_listener

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46250. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44166

[jira] [Resolved] (SPARK-46243) Describe arguments of decode()

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46243. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44157

[jira] [Created] (SPARK-46252) Improve test coverage of memory_profiler.py

2023-12-04 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46252: Summary: Improve test coverage of memory_profiler.py Key: SPARK-46252 URL: https://issues.apache.org/jira/browse/SPARK-46252 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46239) Hide Jetty info

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46239: -- Affects Version/s: 3.4.1 3.3.2 3.2.4

[jira] [Resolved] (SPARK-46239) Spark jetty exposes version information

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46239. --- Fix Version/s: 3.3.4 3.4.3 3.5.1

[jira] [Assigned] (SPARK-46239) Spark jetty exposes version information

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46239: - Assignee: chenyu > Spark jetty exposes version information >

[jira] [Updated] (SPARK-46239) Hide Jetty info

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46239: -- Summary: Hide Jetty info (was: Spark jetty exposes version information) > Hide Jetty info

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders built using `Encoders.tuple` do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Description: In Spark {{3.3.2}} encoders created using {{Encoders.tuple(encoder1, encoder2,

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders built using `Encoders.tuple` do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Description: In Spark {{3.3.2}} encoders created using {{Encoders.tuple(encoder1, encoder2,

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders built using Encoders.tuple do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Summary: Spark 3.3.3 tuple encoders built using Encoders.tuple do not correctly cast null into

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders built using `Encoders.tuple` do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Description: In Spark {{3.3.2}} encoders created using {{Encoders.tuple(encoder1, encoder2,

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders built using `Encoders.tuple` do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Description: In Spark {{3.3.2}} encoders created using {{Encoders.tuple(encoder1, encoder2,

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders built using `Encoders.tuple` do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Description: In Spark {{3.3.2}} encoders created using {{Encoders.tuple(encoder1, encoder2,

[jira] [Updated] (SPARK-46250) Deflake test_parity_listener

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46250: --- Labels: pull-request-available (was: ) > Deflake test_parity_listener >

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders built using `Encoders.tuple` do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Summary: Spark 3.3.3 tuple encoders built using `Encoders.tuple` do not correctly cast null

[jira] [Updated] (SPARK-46251) Spark 3.3.3 tuple encoders do not correctly cast null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Boulter updated SPARK-46251: - Summary: Spark 3.3.3 tuple encoders do not correctly cast null into None for Option values

[jira] [Created] (SPARK-46251) Spark 3.3.3 tuple encoders do not correctly casting null into None for Option values

2023-12-04 Thread Will Boulter (Jira)
Will Boulter created SPARK-46251: Summary: Spark 3.3.3 tuple encoders do not correctly casting null into None for Option values Key: SPARK-46251 URL: https://issues.apache.org/jira/browse/SPARK-46251

[jira] [Created] (SPARK-46250) Deflake test_parity_listener

2023-12-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-46250: --- Summary: Deflake test_parity_listener Key: SPARK-46250 URL: https://issues.apache.org/jira/browse/SPARK-46250 Project: Spark Issue Type: Task Components:

[jira] [Commented] (SPARK-46249) Fix state store metrics access after commit

2023-12-04 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793036#comment-17793036 ] Anish Shrigondekar commented on SPARK-46249: PR here -

[jira] [Updated] (SPARK-46249) Fix state store metrics access after commit

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46249: --- Labels: pull-request-available (was: ) > Fix state store metrics access after commit >

[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46092: -- Fix Version/s: 3.3.4 > Overflow in Parquet row group filter creation causes incorrect results

[jira] [Assigned] (SPARK-46245) Replcace `s.c.MapOps.view.filterKeys` with `s.c.MapOps.filter`

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46245: - Assignee: Yang Jie > Replcace `s.c.MapOps.view.filterKeys` with `s.c.MapOps.filter` >

[jira] [Resolved] (SPARK-46245) Replcace `s.c.MapOps.view.filterKeys` with `s.c.MapOps.filter`

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46245. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44160

[jira] [Assigned] (SPARK-32246) Enable streaming-kinesis-asl tests in Github Action CI

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32246: - Assignee: junyuc25 > Enable streaming-kinesis-asl tests in Github Action CI >

[jira] [Updated] (SPARK-32246) Enable streaming-kinesis-asl tests in Github Action CI

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32246: -- Summary: Enable streaming-kinesis-asl tests in Github Action CI (was: Have a way to

[jira] [Resolved] (SPARK-32246) Have a way to optionally run streaming-kinesis-asl

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32246. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43736

[jira] (SPARK-38473) Use error classes in org.apache.spark.scheduler

2023-12-04 Thread Asmita Limaye (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38473 ] Asmita Limaye deleted comment on SPARK-38473: --- was (Author: JIRAUSER303226): Hi [~bozhang], I was working on this issue (with Hannah) and had some doubts related to how to assign the

[jira] [Created] (SPARK-46249) Fix state store metrics access after commit

2023-12-04 Thread Anish Shrigondekar (Jira)
Anish Shrigondekar created SPARK-46249: -- Summary: Fix state store metrics access after commit Key: SPARK-46249 URL: https://issues.apache.org/jira/browse/SPARK-46249 Project: Spark

[jira] [Updated] (SPARK-46248) Support ignoreCorruptFiles and ignoreMissingFiles options in XML

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46248: --- Labels: pull-request-available (was: ) > Support ignoreCorruptFiles and ignoreMissingFiles

[jira] [Created] (SPARK-46248) Support ignoreCorruptFiles and ignoreMissingFiles options in XML

2023-12-04 Thread Shujing Yang (Jira)
Shujing Yang created SPARK-46248: Summary: Support ignoreCorruptFiles and ignoreMissingFiles options in XML Key: SPARK-46248 URL: https://issues.apache.org/jira/browse/SPARK-46248 Project: Spark

[jira] [Updated] (SPARK-39800) DataSourceV2: View support

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-39800: --- Labels: pull-request-available (was: ) > DataSourceV2: View support >

[jira] [Updated] (SPARK-46225) Collapse stacked withColumns into a single message

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46225: --- Labels: pull-request-available (was: ) > Collapse stacked withColumns into a single

[jira] [Updated] (SPARK-46246) Support EXECUTE IMMEDIATE sytax in Spark SQL

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46246: --- Labels: pull-request-available (was: ) > Support EXECUTE IMMEDIATE sytax in Spark SQL >

[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46092: -- Fix Version/s: 3.4.3 > Overflow in Parquet row group filter creation causes incorrect results

[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46092: -- Fix Version/s: 3.5.1 > Overflow in Parquet row group filter creation causes incorrect results

[jira] [Resolved] (SPARK-46231) Migrate all remaining NotImplementedError & TypeError into PySpark error framework

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46231. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44148

[jira] [Assigned] (SPARK-46231) Migrate all remaining NotImplementedError & TypeError into PySpark error framework

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46231: - Assignee: Haejoon Lee > Migrate all remaining NotImplementedError & TypeError into

[jira] [Updated] (SPARK-46237) Make `HiveDDLSuite` independently testable

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46237: -- Summary: Make `HiveDDLSuite` independently testable (was: Fix test failed of `HiveDDLSuite`)

[jira] [Assigned] (SPARK-46237) Fix test failed of `HiveDDLSuite`

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46237: - Assignee: Yang Jie > Fix test failed of `HiveDDLSuite` >

[jira] [Resolved] (SPARK-46237) Fix test failed of `HiveDDLSuite`

2023-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46237. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44153

[jira] [Commented] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-12-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792942#comment-17792942 ] Bruce Robbins commented on SPARK-45644: --- Even though this is the original issue, I closed it as a

[jira] [Resolved] (SPARK-45644) After upgrading to Spark 3.4.1 and 3.5.0 we receive RuntimeException "scala.Some is not a valid external type for schema of array"

2023-12-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-45644. --- Resolution: Duplicate > After upgrading to Spark 3.4.1 and 3.5.0 we receive

[jira] [Created] (SPARK-46247) Invalid bucket file error when reading from bucketed table created with PathOutputCommitProtocol

2023-12-04 Thread Jira
Никита Соколов created SPARK-46247: -- Summary: Invalid bucket file error when reading from bucketed table created with PathOutputCommitProtocol Key: SPARK-46247 URL:

[jira] [Created] (SPARK-46246) Support EXECUTE IMMEDIATE sytax in Spark SQL

2023-12-04 Thread Milan Stefanovic (Jira)
Milan Stefanovic created SPARK-46246: Summary: Support EXECUTE IMMEDIATE sytax in Spark SQL Key: SPARK-46246 URL: https://issues.apache.org/jira/browse/SPARK-46246 Project: Spark Issue

[jira] [Commented] (SPARK-44900) Cached DataFrame keeps growing

2023-12-04 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792872#comment-17792872 ] 王范明 commented on SPARK-44900: - I have analyzed the program execution details in the logs and it seems that

[jira] [Updated] (SPARK-46245) Replcace `s.c.MapOps.view.filterKeys` with `s.c.MapOps.filter`

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46245: --- Labels: pull-request-available (was: ) > Replcace `s.c.MapOps.view.filterKeys` with

[jira] [Created] (SPARK-46245) Replcace `s.c.MapOps.view.filterKeys` with `s.c.MapOps.filter`

2023-12-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-46245: Summary: Replcace `s.c.MapOps.view.filterKeys` with `s.c.MapOps.filter` Key: SPARK-46245 URL: https://issues.apache.org/jira/browse/SPARK-46245 Project: Spark

[jira] [Updated] (SPARK-46244) INSERT/UPDATE * in MERGE should follow the same semantic of INSERT BY NAME

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46244: --- Labels: pull-request-available (was: ) > INSERT/UPDATE * in MERGE should follow the same

[jira] [Created] (SPARK-46244) INSERT/UPDATE * in MERGE should follow the same semantic of INSERT BY NAME

2023-12-04 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-46244: --- Summary: INSERT/UPDATE * in MERGE should follow the same semantic of INSERT BY NAME Key: SPARK-46244 URL: https://issues.apache.org/jira/browse/SPARK-46244 Project:

[jira] [Updated] (SPARK-46243) Describe arguments of decode()

2023-12-04 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-46243: - Description: Update the description of the `StringDecode` expression and apparently the `decode()`

[jira] [Created] (SPARK-46243) Describe arguments of decode()

2023-12-04 Thread Max Gekk (Jira)
Max Gekk created SPARK-46243: Summary: Describe arguments of decode() Key: SPARK-46243 URL: https://issues.apache.org/jira/browse/SPARK-46243 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-46243) Describe arguments of decode()

2023-12-04 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-46243: - Fix Version/s: (was: 4.0.0) > Describe arguments of decode() > -- > >

[jira] [Resolved] (SPARK-46234) Introduce PySparkKeyError for PySpark error framework

2023-12-04 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-46234. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44151

[jira] [Assigned] (SPARK-46234) Introduce PySparkKeyError for PySpark error framework

2023-12-04 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-46234: - Assignee: Haejoon Lee > Introduce PySparkKeyError for PySpark error framework >

[jira] [Created] (SPARK-46242) Enable all Kinesis tests in Github Actions by default

2023-12-04 Thread junyuc25 (Jira)
junyuc25 created SPARK-46242: Summary: Enable all Kinesis tests in Github Actions by default Key: SPARK-46242 URL: https://issues.apache.org/jira/browse/SPARK-46242 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46241) Fix error handling routine so it wouldn't fall into infinite recursion

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46241: --- Labels: pull-request-available (was: ) > Fix error handling routine so it wouldn't fall

[jira] [Resolved] (SPARK-46186) Invalid Spark Connect execution state transition if interrupted before thread started

2023-12-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46186. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44095

[jira] [Created] (SPARK-46241) Fix error handling routine so it wouldn't fall into infinite recursion

2023-12-04 Thread Alice Sayutina (Jira)
Alice Sayutina created SPARK-46241: -- Summary: Fix error handling routine so it wouldn't fall into infinite recursion Key: SPARK-46241 URL: https://issues.apache.org/jira/browse/SPARK-46241 Project:

[jira] [Created] (SPARK-46240) Add PrepExecutedPlanRule to SparkSessionExtensions

2023-12-04 Thread jiang13021 (Jira)
jiang13021 created SPARK-46240: -- Summary: Add PrepExecutedPlanRule to SparkSessionExtensions Key: SPARK-46240 URL: https://issues.apache.org/jira/browse/SPARK-46240 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46240) Add PrepExecutedPlanRule to SparkSessionExtensions

2023-12-04 Thread jiang13021 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiang13021 updated SPARK-46240: --- Description: Some rules (Rule[SparkPlan]) are applied when preparing for the executedPlan.

[jira] [Updated] (SPARK-46239) Spark jetty exposes version information

2023-12-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46239: --- Labels: pull-request-available (was: ) > Spark jetty exposes version information >

[jira] [Commented] (SPARK-46239) Spark jetty exposes version information

2023-12-04 Thread chenyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792746#comment-17792746 ] chenyu commented on SPARK-46239: It is unsafe to expose version information. It will obtain remote WWW

  1   2   >