[jira] [Commented] (SPARK-41157) Show detailed differences in test

[ https://issues.apache.org/jira/browse/SPARK-41157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634687#comment-17634687 ] Apache Spark commented on SPARK-41157: -- User 'zhengruifeng' has created a pull request for this

[jira] [Updated] (SPARK-41157) Show detailed differences in dataframe comparison

[ https://issues.apache.org/jira/browse/SPARK-41157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-41157: -- Summary: Show detailed differences in dataframe comparison (was: Show detailed differences

[jira] [Assigned] (SPARK-41157) Show detailed differences in test

[ https://issues.apache.org/jira/browse/SPARK-41157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41157: Assignee: (was: Apache Spark) > Show detailed differences in test >

[jira] [Assigned] (SPARK-41157) Show detailed differences in test

[ https://issues.apache.org/jira/browse/SPARK-41157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41157: Assignee: Apache Spark > Show detailed differences in test >

[jira] [Commented] (SPARK-41157) Show detailed differences in test

[ https://issues.apache.org/jira/browse/SPARK-41157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634686#comment-17634686 ] Apache Spark commented on SPARK-41157: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-41157) Show detailed differences in test

Ruifeng Zheng created SPARK-41157: - Summary: Show detailed differences in test Key: SPARK-41157 URL: https://issues.apache.org/jira/browse/SPARK-41157 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-41156) Remove the class TypeCheckFailure

[ https://issues.apache.org/jira/browse/SPARK-41156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634674#comment-17634674 ] Apache Spark commented on SPARK-41156: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-41156) Remove the class TypeCheckFailure

[ https://issues.apache.org/jira/browse/SPARK-41156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41156: Assignee: Max Gekk (was: Apache Spark) > Remove the class TypeCheckFailure >

[jira] [Assigned] (SPARK-41156) Remove the class TypeCheckFailure

[ https://issues.apache.org/jira/browse/SPARK-41156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41156: Assignee: Apache Spark (was: Max Gekk) > Remove the class TypeCheckFailure >

[jira] [Created] (SPARK-41156) Remove the class TypeCheckFailure

Max Gekk created SPARK-41156: Summary: Remove the class TypeCheckFailure Key: SPARK-41156 URL: https://issues.apache.org/jira/browse/SPARK-41156 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-39370) Inline type hints in PySpark

[ https://issues.apache.org/jira/browse/SPARK-39370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-39370: - Description: * SPARK-36845 Inline type hint files for files in python/pyspark/sql * SPARK-37093

[jira] [Assigned] (SPARK-41155) Add error message to SchemaColumnConvertNotSupportedException

[ https://issues.apache.org/jira/browse/SPARK-41155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41155: Assignee: Apache Spark (was: L. C. Hsieh) > Add error message to

[jira] [Commented] (SPARK-41155) Add error message to SchemaColumnConvertNotSupportedException

[ https://issues.apache.org/jira/browse/SPARK-41155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634646#comment-17634646 ] Apache Spark commented on SPARK-41155: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-41155) Add error message to SchemaColumnConvertNotSupportedException

[ https://issues.apache.org/jira/browse/SPARK-41155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41155: Assignee: L. C. Hsieh (was: Apache Spark) > Add error message to

[jira] [Created] (SPARK-41155) Add error message to SchemaColumnConvertNotSupportedException

L. C. Hsieh created SPARK-41155: --- Summary: Add error message to SchemaColumnConvertNotSupportedException Key: SPARK-41155 URL: https://issues.apache.org/jira/browse/SPARK-41155 Project: Spark

[jira] [Created] (SPARK-41154) Incorrect relation caching for queries with time travel spec

Allison Portis created SPARK-41154: -- Summary: Incorrect relation caching for queries with time travel spec Key: SPARK-41154 URL: https://issues.apache.org/jira/browse/SPARK-41154 Project: Spark

[jira] [Commented] (SPARK-41153) Log migrated shuffle data size and migration time

[ https://issues.apache.org/jira/browse/SPARK-41153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634639#comment-17634639 ] Apache Spark commented on SPARK-41153: -- User 'warrenzhu25' has created a pull request for this

[jira] [Assigned] (SPARK-41153) Log migrated shuffle data size and migration time

[ https://issues.apache.org/jira/browse/SPARK-41153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41153: Assignee: Apache Spark > Log migrated shuffle data size and migration time >

[jira] [Commented] (SPARK-41153) Log migrated shuffle data size and migration time

[ https://issues.apache.org/jira/browse/SPARK-41153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634640#comment-17634640 ] Apache Spark commented on SPARK-41153: -- User 'warrenzhu25' has created a pull request for this

[jira] [Assigned] (SPARK-41153) Log migrated shuffle data size and migration time

[ https://issues.apache.org/jira/browse/SPARK-41153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41153: Assignee: (was: Apache Spark) > Log migrated shuffle data size and migration time >

[jira] [Created] (SPARK-41153) Log migrated shuffle data size and migration time

Zhongwei Zhu created SPARK-41153: Summary: Log migrated shuffle data size and migration time Key: SPARK-41153 URL: https://issues.apache.org/jira/browse/SPARK-41153 Project: Spark Issue

[jira] [Created] (SPARK-41152) Print out schema as tree string format for state store schema checker

Jungtaek Lim created SPARK-41152: Summary: Print out schema as tree string format for state store schema checker Key: SPARK-41152 URL: https://issues.apache.org/jira/browse/SPARK-41152 Project: Spark

[jira] [Created] (SPARK-41151) Keep built-in file _metadata column nullable value consistent

Yaohua Zhao created SPARK-41151: --- Summary: Keep built-in file _metadata column nullable value consistent Key: SPARK-41151 URL: https://issues.apache.org/jira/browse/SPARK-41151 Project: Spark

[jira] [Commented] (SPARK-40798) Alter partition should verify value

[ https://issues.apache.org/jira/browse/SPARK-40798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634614#comment-17634614 ] Apache Spark commented on SPARK-40798: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Resolved] (SPARK-40622) Result of a single task in collect() must fit in 2GB

[ https://issues.apache.org/jira/browse/SPARK-40622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40622. - Fix Version/s: 3.4.0 Resolution: Fixed > Result of a single task in

[jira] [Assigned] (SPARK-40622) Result of a single task in collect() must fit in 2GB

[ https://issues.apache.org/jira/browse/SPARK-40622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40622: --- Assignee: Ziqi Liu > Result of a single task in collect() must fit in 2GB

[jira] [Resolved] (SPARK-38497) Improve the test coverage for pyspark/*.py

[ https://issues.apache.org/jira/browse/SPARK-38497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-38497. - Resolution: Duplicate This is done by other several sub-tasks, such as SPARK-38854 SPARK-38879

[jira] [Updated] (SPARK-41150) Document debugging with PySpark memory profiler

[ https://issues.apache.org/jira/browse/SPARK-41150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41150: - Description: Document how to debug with PySpark memory profiler on

[jira] [Updated] (SPARK-41150) Document debugging with PySpark memory profiler

[ https://issues.apache.org/jira/browse/SPARK-41150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41150: - Summary: Document debugging with PySpark memory profiler (was: Document PySpark memory

[jira] [Created] (SPARK-41150) Document PySpark memory profiler

Xinrong Meng created SPARK-41150: Summary: Document PySpark memory profiler Key: SPARK-41150 URL: https://issues.apache.org/jira/browse/SPARK-41150 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-41149) Fix `SparkSession.builder.config` to support other values

[ https://issues.apache.org/jira/browse/SPARK-41149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634589#comment-17634589 ] Haejoon Lee commented on SPARK-41149: - I'm working on it > Fix `SparkSession.builder.config` to

[jira] [Created] (SPARK-41149) Fix `SparkSession.builder.config` to support other values

Haejoon Lee created SPARK-41149: --- Summary: Fix `SparkSession.builder.config` to support other values Key: SPARK-41149 URL: https://issues.apache.org/jira/browse/SPARK-41149 Project: Spark

[jira] [Resolved] (SPARK-41113) Upgrade sbt to 1.8.0

[ https://issues.apache.org/jira/browse/SPARK-41113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41113. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38620

[jira] [Assigned] (SPARK-41113) Upgrade sbt to 1.8.0

[ https://issues.apache.org/jira/browse/SPARK-41113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41113: - Assignee: Yang Jie > Upgrade sbt to 1.8.0 > > >

[jira] [Resolved] (SPARK-40755) Migrate type check failures of number formatting onto error classes

[ https://issues.apache.org/jira/browse/SPARK-40755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40755. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38531

[jira] [Assigned] (SPARK-40755) Migrate type check failures of number formatting onto error classes

[ https://issues.apache.org/jira/browse/SPARK-40755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40755: Assignee: BingKun Pan > Migrate type check failures of number formatting onto error classes >

[jira] [Comment Edited] (SPARK-39865) Show proper error messages on the overflow errors of table insert

[ https://issues.apache.org/jira/browse/SPARK-39865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634465#comment-17634465 ] Catalin Toda edited comment on SPARK-39865 at 11/15/22 6:02 PM:

[jira] [Commented] (SPARK-39865) Show proper error messages on the overflow errors of table insert

[ https://issues.apache.org/jira/browse/SPARK-39865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17634465#comment-17634465 ] Catalin Toda commented on SPARK-39865: -- [~Gengliang.Wang] here is the SQL statements that fail in

[jira] [Resolved] (SPARK-40519) Add "Publish workflow" to help release apache/spark image

[ https://issues.apache.org/jira/browse/SPARK-40519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-40519. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 23

[jira] [Created] (SPARK-41148) Implement `DataFrame.dropna ` and `DataFrame.na.drop `

Ruifeng Zheng created SPARK-41148: - Summary: Implement `DataFrame.dropna ` and `DataFrame.na.drop ` Key: SPARK-41148 URL: https://issues.apache.org/jira/browse/SPARK-41148 Project: Spark

[jira] [Resolved] (SPARK-41140) Assign a name to the legacy error class _LEGACY_ERROR_TEMP_2440

[ https://issues.apache.org/jira/browse/SPARK-41140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-41140. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38656

[jira] [Resolved] (SPARK-41128) Implement `DataFrame.fillna ` and `DataFrame.na.fill `

[ https://issues.apache.org/jira/browse/SPARK-41128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-41128. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38653

[jira] [Assigned] (SPARK-40956) SQL Equivalent for Dataframe overwrite command

[ https://issues.apache.org/jira/browse/SPARK-40956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40956: --- Assignee: chengyan fu > SQL Equivalent for Dataframe overwrite command >

[jira] [Resolved] (SPARK-40956) SQL Equivalent for Dataframe overwrite command

[ https://issues.apache.org/jira/browse/SPARK-40956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40956. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38404

[jira] [Assigned] (SPARK-41144) UnresolvedHint should not cause query failure

[ https://issues.apache.org/jira/browse/SPARK-41144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-41144: --- Assignee: XiDuo You > UnresolvedHint should not cause query failure >

[jira] [Resolved] (SPARK-41144) UnresolvedHint should not cause query failure

[ https://issues.apache.org/jira/browse/SPARK-41144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-41144. - Fix Version/s: 3.3.2 3.4.0 Resolution: Fixed Issue resolved by pull

[jira] [Resolved] (SPARK-41112) RuntimeFilter should apply ColumnPruning eagerly with in-subquery filter

[ https://issues.apache.org/jira/browse/SPARK-41112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-41112. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 38619

[jira] [Assigned] (SPARK-41112) RuntimeFilter should apply ColumnPruning eagerly with in-subquery filter

[ https://issues.apache.org/jira/browse/SPARK-41112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-41112: --- Assignee: XiDuo You > RuntimeFilter should apply ColumnPruning eagerly with in-subquery