[jira] [Created] (SPARK-45027) Hide internal functions/variables in `pyspark.sql.functions` from auto-completion

Ruifeng Zheng created SPARK-45027: - Summary: Hide internal functions/variables in `pyspark.sql.functions` from auto-completion Key: SPARK-45027 URL: https://issues.apache.org/jira/browse/SPARK-45027

[jira] [Assigned] (SPARK-45024) Filter out some configs in Session Creation

[ https://issues.apache.org/jira/browse/SPARK-45024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45024: - Assignee: Ruifeng Zheng > Filter out some configs in Session Creation >

[jira] [Resolved] (SPARK-45024) Filter out some configs in Session Creation

[ https://issues.apache.org/jira/browse/SPARK-45024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45024. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42741

[jira] [Commented] (SPARK-45024) Filter out some configs in Session Creation

[ https://issues.apache.org/jira/browse/SPARK-45024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760710#comment-17760710 ] Snoot.io commented on SPARK-45024: -- User 'zhengruifeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-45024) Filter out some configs in Session Creation

[ https://issues.apache.org/jira/browse/SPARK-45024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760709#comment-17760709 ] Snoot.io commented on SPARK-45024: -- User 'zhengruifeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760708#comment-17760708 ] Snoot.io commented on SPARK-44940: -- User 'sadikovi' has created a pull request for this issue:

[jira] [Commented] (SPARK-45018) Add CalendarIntervalType to Python Client

[ https://issues.apache.org/jira/browse/SPARK-45018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760704#comment-17760704 ] Snoot.io commented on SPARK-45018: -- User 'zhengruifeng' has created a pull request for this issue:

[jira] [Commented] (SPARK-44900) Cached DataFrame keeps growing

[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760700#comment-17760700 ] Varun Nalla commented on SPARK-44900: - [~yxzhang] / [~yao] any update for us ? > Cached DataFrame

[jira] [Created] (SPARK-45026) non-command spark.sql should support datatypes not compatible with arrow

Ruifeng Zheng created SPARK-45026: - Summary: non-command spark.sql should support datatypes not compatible with arrow Key: SPARK-45026 URL: https://issues.apache.org/jira/browse/SPARK-45026 Project:

[jira] [Updated] (SPARK-45026) spark.sql should support datatypes not compatible with arrow

[ https://issues.apache.org/jira/browse/SPARK-45026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-45026: -- Summary: spark.sql should support datatypes not compatible with arrow (was: non-command

[jira] [Resolved] (SPARK-45012) CheckAnalysis should throw inlined plan in AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-45012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45012. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42729

[jira] [Commented] (SPARK-45025) Block manager write to memory store iterator should process thread interrupt

[ https://issues.apache.org/jira/browse/SPARK-45025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760677#comment-17760677 ] Anish Shrigondekar commented on SPARK-45025: cc - [~kabhwan] - PR here -

[jira] [Created] (SPARK-45025) Block manager write to memory store iterator should process thread interrupt

Anish Shrigondekar created SPARK-45025: -- Summary: Block manager write to memory store iterator should process thread interrupt Key: SPARK-45025 URL: https://issues.apache.org/jira/browse/SPARK-45025

[jira] [Created] (SPARK-45024) Filter out some configs in Session Creation

Ruifeng Zheng created SPARK-45024: - Summary: Filter out some configs in Session Creation Key: SPARK-45024 URL: https://issues.apache.org/jira/browse/SPARK-45024 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-45015) Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}`

[ https://issues.apache.org/jira/browse/SPARK-45015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45015: - Assignee: Ruifeng Zheng > Refine DocStrings of `try_{add, subtract, multiply, divide,

[jira] [Resolved] (SPARK-45015) Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}`

[ https://issues.apache.org/jira/browse/SPARK-45015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45015. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42735

[jira] [Resolved] (SPARK-44971) [BUG Fix] PySpark StreamingQuerProgress fromJson

[ https://issues.apache.org/jira/browse/SPARK-44971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44971. -- Fix Version/s: 3.5.1 Assignee: Wei Liu Resolution: Fixed Fixed in

[jira] [Commented] (SPARK-45014) Clean up fileserver when cleaning up files, jars and archives in SparkContext

[ https://issues.apache.org/jira/browse/SPARK-45014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760658#comment-17760658 ] GridGain Integration commented on SPARK-45014: -- User 'HyukjinKwon' has created a pull

[jira] [Commented] (SPARK-45014) Clean up fileserver when cleaning up files, jars and archives in SparkContext

[ https://issues.apache.org/jira/browse/SPARK-45014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760657#comment-17760657 ] Ignite TC Bot commented on SPARK-45014: --- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-45023) SPIP: Python Stored Procedures

[ https://issues.apache.org/jira/browse/SPARK-45023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-45023: - Shepherd: Hyukjin Kwon > SPIP: Python Stored Procedures > -- > >

[jira] [Created] (SPARK-45023) SPIP: Python Stored Procedures

Allison Wang created SPARK-45023: Summary: SPIP: Python Stored Procedures Key: SPARK-45023 URL: https://issues.apache.org/jira/browse/SPARK-45023 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-43299) JVM Client throw StreamingQueryException when error handling is implemented

[ https://issues.apache.org/jira/browse/SPARK-43299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760620#comment-17760620 ] Yihong He commented on SPARK-43299: --- [~hvanhovell] Thanks for the reminder! I will make sure it works.

[jira] [Updated] (SPARK-44991) Spark json schema inference and fromJson api having inconsistent behavior

[ https://issues.apache.org/jira/browse/SPARK-44991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nirav patel updated SPARK-44991: Description: Spark json reader can infer datatype of a fields. I am ingesting millions of

[jira] [Updated] (SPARK-45012) CheckAnalysis should throw inlined plan in AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-45012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-45012: - Affects Version/s: 4.0.0 (was: 3.5.0) > CheckAnalysis should throw inlined

[jira] [Assigned] (SPARK-45016) Add missing `try_remote_functions` annotations

[ https://issues.apache.org/jira/browse/SPARK-45016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45016: - Assignee: Ruifeng Zheng > Add missing `try_remote_functions` annotations >

[jira] [Resolved] (SPARK-45016) Add missing `try_remote_functions` annotations

[ https://issues.apache.org/jira/browse/SPARK-45016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45016. --- Fix Version/s: 3.5.1 Resolution: Fixed Issue resolved by pull request 42734

[jira] [Assigned] (SPARK-45017) Add CalendarIntervalType to PySpark

[ https://issues.apache.org/jira/browse/SPARK-45017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45017: - Assignee: Ruifeng Zheng > Add CalendarIntervalType to PySpark >

[jira] [Resolved] (SPARK-45017) Add CalendarIntervalType to PySpark

[ https://issues.apache.org/jira/browse/SPARK-45017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45017. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42736

[jira] [Resolved] (SPARK-44997) Align example order (Python -> Scala/Java -> R) in all Spark Doc Content

[ https://issues.apache.org/jira/browse/SPARK-44997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44997. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42712

[jira] [Assigned] (SPARK-44997) Align example order (Python -> Scala/Java -> R) in all Spark Doc Content

[ https://issues.apache.org/jira/browse/SPARK-44997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44997: - Assignee: BingKun Pan > Align example order (Python -> Scala/Java -> R) in all Spark

[jira] [Assigned] (SPARK-42304) Assign name to _LEGACY_ERROR_TEMP_2189

[ https://issues.apache.org/jira/browse/SPARK-42304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-42304: - Assignee: Valentin > Assign name to _LEGACY_ERROR_TEMP_2189 >

[jira] [Updated] (SPARK-45005) Reducing the CI time for slow pyspark-pandas-connect tests

[ https://issues.apache.org/jira/browse/SPARK-45005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45005: -- Issue Type: Test (was: Bug) > Reducing the CI time for slow pyspark-pandas-connect tests >

[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dipayan Dev updated SPARK-44884: Priority: Minor (was: Critical) > Spark doesn't create SUCCESS file in Spark 3.3.0+ when

[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dipayan Dev updated SPARK-44884: Priority: Major (was: Minor) > Spark doesn't create SUCCESS file in Spark 3.3.0+ when

[jira] [Resolved] (SPARK-45005) Reducing the CI time for slow pyspark-pandas-connect tests

[ https://issues.apache.org/jira/browse/SPARK-45005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45005. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42719

[jira] [Assigned] (SPARK-45005) Reducing the CI time for slow pyspark-pandas-connect tests

[ https://issues.apache.org/jira/browse/SPARK-45005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45005: - Assignee: Haejoon Lee > Reducing the CI time for slow pyspark-pandas-connect tests >

[jira] [Resolved] (SPARK-45021) Remove `antlr4-maven-plugin` configuration from `sql/catalyst/pom.xml`

[ https://issues.apache.org/jira/browse/SPARK-45021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45021. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42739

[jira] [Assigned] (SPARK-45021) Remove `antlr4-maven-plugin` configuration from `sql/catalyst/pom.xml`

[ https://issues.apache.org/jira/browse/SPARK-45021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45021: - Assignee: Yang Jie > Remove `antlr4-maven-plugin` configuration from

[jira] [Created] (SPARK-45022) Provide context for dataset API errors

Peter Toth created SPARK-45022: -- Summary: Provide context for dataset API errors Key: SPARK-45022 URL: https://issues.apache.org/jira/browse/SPARK-45022 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-44239) Free memory allocated by large vectors when vectors are reset

[ https://issues.apache.org/jira/browse/SPARK-44239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44239. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41782

[jira] [Assigned] (SPARK-44239) Free memory allocated by large vectors when vectors are reset

[ https://issues.apache.org/jira/browse/SPARK-44239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44239: --- Assignee: Wan Kun > Free memory allocated by large vectors when vectors are reset >

[jira] [Commented] (SPARK-45021) Remove `antlr4-maven-plugin` configuration from `sql/catalyst/pom.xml`

[ https://issues.apache.org/jira/browse/SPARK-45021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760404#comment-17760404 ] Ignite TC Bot commented on SPARK-45021: --- User 'LuciferYang' has created a pull request for this

[jira] [Created] (SPARK-45021) Remove `antlr4-maven-plugin` configuration from `sql/catalyst/pom.xml`

Yang Jie created SPARK-45021: Summary: Remove `antlr4-maven-plugin` configuration from `sql/catalyst/pom.xml` Key: SPARK-45021 URL: https://issues.apache.org/jira/browse/SPARK-45021 Project: Spark

[jira] [Commented] (SPARK-45019) Make workflow scala213 on container & clean env

[ https://issues.apache.org/jira/browse/SPARK-45019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760363#comment-17760363 ] Hudson commented on SPARK-45019: User 'panbingkun' has created a pull request for this issue:

[jira] [Created] (SPARK-45020) org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'default' not found (state=08S01,code=0)

Sruthi Mooriyathvariam created SPARK-45020: -- Summary: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'default' not found (state=08S01,code=0) Key: SPARK-45020 URL:

[jira] [Created] (SPARK-45019) Make workflow scala213 on container & clean env

BingKun Pan created SPARK-45019: --- Summary: Make workflow scala213 on container & clean env Key: SPARK-45019 URL: https://issues.apache.org/jira/browse/SPARK-45019 Project: Spark Issue Type:

[jira] [Created] (SPARK-45018) Add CalendarIntervalType to Python Client

Ruifeng Zheng created SPARK-45018: - Summary: Add CalendarIntervalType to Python Client Key: SPARK-45018 URL: https://issues.apache.org/jira/browse/SPARK-45018 Project: Spark Issue Type: New

[jira] [Created] (SPARK-45017) Add CalendarIntervalType to PySpark

Ruifeng Zheng created SPARK-45017: - Summary: Add CalendarIntervalType to PySpark Key: SPARK-45017 URL: https://issues.apache.org/jira/browse/SPARK-45017 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-45015) Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}`

[ https://issues.apache.org/jira/browse/SPARK-45015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760302#comment-17760302 ] ASF GitHub Bot commented on SPARK-45015: User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-45015) Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}`

[ https://issues.apache.org/jira/browse/SPARK-45015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760301#comment-17760301 ] ASF GitHub Bot commented on SPARK-45015: User 'zhengruifeng' has created a pull request for this

[jira] [Updated] (SPARK-45015) Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}`

[ https://issues.apache.org/jira/browse/SPARK-45015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-45015: -- Summary: Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}` (was: Refine

[jira] [Commented] (SPARK-45016) Add missing `try_remote_functions` annotations

[ https://issues.apache.org/jira/browse/SPARK-45016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760298#comment-17760298 ] ASF GitHub Bot commented on SPARK-45016: User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-45005) Reducing the CI time for slow pyspark-pandas-connect tests

[ https://issues.apache.org/jira/browse/SPARK-45005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760291#comment-17760291 ] ASF GitHub Bot commented on SPARK-45005: User 'itholic' has created a pull request for this

[jira] [Created] (SPARK-45016) Add missing `try_remote_functions` annotations

Ruifeng Zheng created SPARK-45016: - Summary: Add missing `try_remote_functions` annotations Key: SPARK-45016 URL: https://issues.apache.org/jira/browse/SPARK-45016 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760281#comment-17760281 ] Maxim Martynov edited comment on SPARK-33628 at 8/30/23 8:53 AM: - Fixed

[jira] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

[ https://issues.apache.org/jira/browse/SPARK-33628 ] Maxim Martynov deleted comment on SPARK-33628: was (Author: JIRAUSER283764): Can anyone review this pull request? > Use the Hive.getPartitionsByNames method instead of

[jira] [Commented] (SPARK-33628) Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

[ https://issues.apache.org/jira/browse/SPARK-33628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760281#comment-17760281 ] Maxim Martynov commented on SPARK-33628: Fixed in SPARK-42480 > Use the

[jira] [Created] (SPARK-45015) Refine DocString of `try_*` functions

Ruifeng Zheng created SPARK-45015: - Summary: Refine DocString of `try_*` functions Key: SPARK-45015 URL: https://issues.apache.org/jira/browse/SPARK-45015 Project: Spark Issue Type: Sub-task