[jira] [Updated] (SPARK-45792) SPIP: ShuffleManager short name registration via SparkPlugin

[ https://issues.apache.org/jira/browse/SPARK-45792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45792: --- Labels: pull-request-available (was: ) > SPIP: ShuffleManager short name registration via S

[jira] [Updated] (SPARK-48551) Perf improvement for escapePathName

[ https://issues.apache.org/jira/browse/SPARK-48551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48551: --- Labels: pull-request-available (was: ) > Perf improvement for escapePathName >

[jira] [Updated] (SPARK-48561) Throw PandasNotImplementedError for unsupported plotting functions

[ https://issues.apache.org/jira/browse/SPARK-48561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48561: --- Labels: pull-request-available (was: ) > Throw PandasNotImplementedError for unsupported pl

[jira] [Commented] (SPARK-48483) Allow UnsafeExternalSorter to spill when other consumer requests memory

[ https://issues.apache.org/jira/browse/SPARK-48483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853004#comment-17853004 ] Ke Jia commented on SPARK-48483: [~cloud_fan] [~ulyssesyou] Do you have any suggestion f

[jira] [Created] (SPARK-48561) Throw PandasNotImplementedError for unsupported plotting functions

Ruifeng Zheng created SPARK-48561: - Summary: Throw PandasNotImplementedError for unsupported plotting functions Key: SPARK-48561 URL: https://issues.apache.org/jira/browse/SPARK-48561 Project: Spark

[jira] [Resolved] (SPARK-48548) Update LICENSE/NOTICE for spark-core with shaded dependencies

[ https://issues.apache.org/jira/browse/SPARK-48548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48548. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46891 [https://github.com

[jira] [Assigned] (SPARK-48548) Update LICENSE/NOTICE for spark-core with shaded dependencies

[ https://issues.apache.org/jira/browse/SPARK-48548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48548: Assignee: Kent Yao > Update LICENSE/NOTICE for spark-core with shaded dependencies >

[jira] [Commented] (SPARK-48286) Analyze 'exists' default expression instead of 'current' default expression in structField to v2 column conversion

[ https://issues.apache.org/jira/browse/SPARK-48286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852984#comment-17852984 ] melin commented on SPARK-48286: --- defaultValueNotConstantError method not exists > Analyze

[jira] [Resolved] (SPARK-48552) multi-line CSV schema inference should also throw FAILED_READ_FILE

[ https://issues.apache.org/jira/browse/SPARK-48552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48552. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46890 [https://gi

[jira] [Assigned] (SPARK-48552) multi-line CSV schema inference should also throw FAILED_READ_FILE

[ https://issues.apache.org/jira/browse/SPARK-48552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48552: Assignee: Wenchen Fan > multi-line CSV schema inference should also throw FAILED_READ_FIL

[jira] [Created] (SPARK-48560) Make StreamingQueryListener.spark settable

Hyukjin Kwon created SPARK-48560: Summary: Make StreamingQueryListener.spark settable Key: SPARK-48560 URL: https://issues.apache.org/jira/browse/SPARK-48560 Project: Spark Issue Type: Improv

[jira] [Assigned] (SPARK-47952) Support retrieving the real SparkConnectService GRPC address and port programmatically when running on Yarn

[ https://issues.apache.org/jira/browse/SPARK-47952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47952: Assignee: TakawaAkirayo (was: Adam Binford) > Support retrieving the real SparkConnectSe

[jira] [Resolved] (SPARK-47952) Support retrieving the real SparkConnectService GRPC address and port programmatically when running on Yarn

[ https://issues.apache.org/jira/browse/SPARK-47952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47952. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46182 [https://gi

[jira] [Assigned] (SPARK-47952) Support retrieving the real SparkConnectService GRPC address and port programmatically when running on Yarn

[ https://issues.apache.org/jira/browse/SPARK-47952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47952: Assignee: Adam Binford > Support retrieving the real SparkConnectService GRPC address and

[jira] [Updated] (SPARK-37448) Multiple optimizations related to CurrentOrigin.withOrigin

[ https://issues.apache.org/jira/browse/SPARK-37448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-37448: --- Labels: pull-request-available (was: ) > Multiple optimizations related to CurrentOrigin.wi

[jira] [Updated] (SPARK-37448) Multiple optimizations related to CurrentOrigin.withOrigin

[ https://issues.apache.org/jira/browse/SPARK-37448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-37448: --- Description: This PR / Jira proposes multiple optimizations related to CurrentOrigin.withOrigin, in

[jira] [Updated] (SPARK-37448) Multiple optimizations related to CurrentOrigin.withOrigin

[ https://issues.apache.org/jira/browse/SPARK-37448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-37448: --- Affects Version/s: 4.0.0 (was: 3.2.0) > Multiple optimizations related to

[jira] [Updated] (SPARK-37448) Multiple optimizations related to CurrentOrigin.withOrigin

[ https://issues.apache.org/jira/browse/SPARK-37448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-37448: --- Summary: Multiple optimizations related to CurrentOrigin.withOrigin (was: Avoid CurrentOrigin.withO

[jira] [Resolved] (SPARK-48553) Cache more properties

[ https://issues.apache.org/jira/browse/SPARK-48553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48553. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46896 [https://

[jira] [Updated] (SPARK-48558) Improve accumulator V2 error message

[ https://issues.apache.org/jira/browse/SPARK-48558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48558: --- Labels: pull-request-available (was: ) > Improve accumulator V2 error message > ---

[jira] [Assigned] (SPARK-48286) Analyze 'exists' default expression instead of 'current' default expression in structField to v2 column conversion

[ https://issues.apache.org/jira/browse/SPARK-48286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48286: --- Assignee: Uros Stankovic > Analyze 'exists' default expression instead of 'current' default

[jira] [Resolved] (SPARK-48286) Analyze 'exists' default expression instead of 'current' default expression in structField to v2 column conversion

[ https://issues.apache.org/jira/browse/SPARK-48286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48286. - Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46594 [https://gith

[jira] [Resolved] (SPARK-48283) Implement modified Lowercase operation for UTF8_BINARY_LCASE

[ https://issues.apache.org/jira/browse/SPARK-48283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48283. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46700 [https://gith

[jira] [Resolved] (SPARK-48435) UNICODE collation should not support binary equality

[ https://issues.apache.org/jira/browse/SPARK-48435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48435. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46772 [https://gith

[jira] [Assigned] (SPARK-48435) UNICODE collation should not support binary equality

[ https://issues.apache.org/jira/browse/SPARK-48435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48435: --- Assignee: Uroš Bojanić > UNICODE collation should not support binary equality > ---

[jira] [Updated] (SPARK-48557) Support scalar subquery with group-by on column equal to constant

[ https://issues.apache.org/jira/browse/SPARK-48557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48557: --- Labels: pull-request-available (was: ) > Support scalar subquery with group-by on column eq

[jira] [Created] (SPARK-48557) Support scalar subquery with group-by on column equal to constant

Jack Chen created SPARK-48557: - Summary: Support scalar subquery with group-by on column equal to constant Key: SPARK-48557 URL: https://issues.apache.org/jira/browse/SPARK-48557 Project: Spark

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Shepherd: Wenchen Fan > Minimizing calls to HiveMetaStore layer for getting partitions, when tables > are repe

[jira] [Assigned] (SPARK-48526) Allow passing custom sink to StreamTest::testStream

[ https://issues.apache.org/jira/browse/SPARK-48526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48526: --- Assignee: Johan Lasperas > Allow passing custom sink to StreamTest::testStream > --

[jira] [Resolved] (SPARK-48526) Allow passing custom sink to StreamTest::testStream

[ https://issues.apache.org/jira/browse/SPARK-48526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48526. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46866 [https://gith

[jira] [Assigned] (SPARK-48546) Fix ExpressionEncoder after replacing NullPointerExceptions with proper error classes in AssertNotNull expression

[ https://issues.apache.org/jira/browse/SPARK-48546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48546: --- Assignee: Daniel > Fix ExpressionEncoder after replacing NullPointerExceptions with proper

[jira] [Resolved] (SPARK-48546) Fix ExpressionEncoder after replacing NullPointerExceptions with proper error classes in AssertNotNull expression

[ https://issues.apache.org/jira/browse/SPARK-48546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48546. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46888 [https://gith

[jira] [Updated] (SPARK-48555) Support Column type for several SQL functions in scala and python

[ https://issues.apache.org/jira/browse/SPARK-48555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48555: --- Labels: pull-request-available (was: ) > Support Column type for several SQL functions in s

[jira] [Updated] (SPARK-48554) Use R 4.4.0 in `windows` R GitHub Action Window job

[ https://issues.apache.org/jira/browse/SPARK-48554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48554: -- Summary: Use R 4.4.0 in `windows` R GitHub Action Window job (was: Use R 4.4.0 in `windows` R

[jira] [Updated] (SPARK-48554) Use R 4.4.0 in `windows` R GitHub Action Windows job

[ https://issues.apache.org/jira/browse/SPARK-48554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48554: -- Summary: Use R 4.4.0 in `windows` R GitHub Action Windows job (was: Use R 4.4.0 in `windows`

[jira] [Updated] (SPARK-48555) Support Column type for several SQL functions in scala and python

[ https://issues.apache.org/jira/browse/SPARK-48555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Serruya updated SPARK-48555: Priority: Major (was: Minor) > Support Column type for several SQL functions in scala and python

[jira] [Updated] (SPARK-48556) Incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION

[ https://issues.apache.org/jira/browse/SPARK-48556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48556: --- Labels: pull-request-available (was: ) > Incorrect error message pointing to UNSUPPORTED_GR

[jira] [Created] (SPARK-48556) Incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION

Nikola Mandic created SPARK-48556: - Summary: Incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION Key: SPARK-48556 URL: https://issues.apache.org/jira/browse/SPARK-48556 Project: Spark

[jira] [Updated] (SPARK-48091) Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored

[ https://issues.apache.org/jira/browse/SPARK-48091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Serruya updated SPARK-48091: Description: When using an `explode` function, and `transform` function in the same select statem

[jira] [Updated] (SPARK-48091) Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored

[ https://issues.apache.org/jira/browse/SPARK-48091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Serruya updated SPARK-48091: Environment: Scala 2.12.15, Python 3.10, 3.12, OSX 14.4 and Databricks DBR 13.3, 14.3, Pyspark 3.4

[jira] [Updated] (SPARK-48091) Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored

[ https://issues.apache.org/jira/browse/SPARK-48091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Serruya updated SPARK-48091: Component/s: Spark Core (was: PySpark) > Using `explode` together with `trans

[jira] [Updated] (SPARK-48091) Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored

[ https://issues.apache.org/jira/browse/SPARK-48091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Serruya updated SPARK-48091: Labels: alias (was: PySpark alias) > Using `explode` together with `transform` in the same select

[jira] [Created] (SPARK-48555) Support Column type for several SQL functions in scala and python

Ron Serruya created SPARK-48555: --- Summary: Support Column type for several SQL functions in scala and python Key: SPARK-48555 URL: https://issues.apache.org/jira/browse/SPARK-48555 Project: Spark

[jira] [Updated] (SPARK-48555) Support Column type for several SQL functions in scala and python

[ https://issues.apache.org/jira/browse/SPARK-48555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Serruya updated SPARK-48555: Description: Currently, several SQL functions accept both native types and Columns, but only acce

[jira] [Resolved] (SPARK-48513) Use NERF framework for state schema compatibility exception

[ https://issues.apache.org/jira/browse/SPARK-48513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48513. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46856 [https://gi

[jira] [Assigned] (SPARK-48513) Use NERF framework for state schema compatibility exception

[ https://issues.apache.org/jira/browse/SPARK-48513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48513: Assignee: Anish Shrigondekar > Use NERF framework for state schema compatibility exceptio

[jira] [Commented] (SPARK-48463) MLLib function unable to handle nested data

[ https://issues.apache.org/jira/browse/SPARK-48463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852751#comment-17852751 ] someshwar kale commented on SPARK-48463: As a temporary fix you may consider ren

[jira] [Updated] (SPARK-48350) [M0] Support for exceptions thrown from parser/interpreter

[ https://issues.apache.org/jira/browse/SPARK-48350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Milicevic updated SPARK-48350: Description: In general, add support for SQL scripting related exceptions. By the time som

[jira] [Updated] (SPARK-48554) Use R 4.4.0 in `windows` R GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-48554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48554: --- Labels: pull-request-available (was: ) > Use R 4.4.0 in `windows` R GitHub Action job > ---

[jira] [Resolved] (SPARK-48550) Directly use the parent Window class

[ https://issues.apache.org/jira/browse/SPARK-48550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48550. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46892 [https://gi

[jira] [Assigned] (SPARK-48550) Directly use the parent Window class

[ https://issues.apache.org/jira/browse/SPARK-48550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48550: Assignee: Ruifeng Zheng > Directly use the parent Window class >

[jira] [Updated] (SPARK-48553) Cache more properties

[ https://issues.apache.org/jira/browse/SPARK-48553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48553: --- Labels: pull-request-available (was: ) > Cache more properties > - > >

[jira] [Created] (SPARK-48553) Cache more properties

Ruifeng Zheng created SPARK-48553: - Summary: Cache more properties Key: SPARK-48553 URL: https://issues.apache.org/jira/browse/SPARK-48553 Project: Spark Issue Type: Sub-task Compon

[jira] [Updated] (SPARK-48552) multi-line CSV schema inference should also throw FAILED_READ_FILE

[ https://issues.apache.org/jira/browse/SPARK-48552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48552: --- Labels: pull-request-available (was: ) > multi-line CSV schema inference should also throw

[jira] [Created] (SPARK-48552) multi-line CSV schema inference should also throw FAILED_READ_FILE

Wenchen Fan created SPARK-48552: --- Summary: multi-line CSV schema inference should also throw FAILED_READ_FILE Key: SPARK-48552 URL: https://issues.apache.org/jira/browse/SPARK-48552 Project: Spark

[jira] [Assigned] (SPARK-48548) Update LICENSE/NOTICE for spark-core with shaded dependencies

[ https://issues.apache.org/jira/browse/SPARK-48548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48548: -- Assignee: (was: Apache Spark) > Update LICENSE/NOTICE for spark-core with shaded

[jira] [Assigned] (SPARK-48548) Update LICENSE/NOTICE for spark-core with shaded dependencies

[ https://issues.apache.org/jira/browse/SPARK-48548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48548: -- Assignee: Apache Spark > Update LICENSE/NOTICE for spark-core with shaded dependencie

[jira] [Assigned] (SPARK-48548) Update LICENSE/NOTICE for spark-core with shaded dependencies

[ https://issues.apache.org/jira/browse/SPARK-48548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48548: -- Assignee: Apache Spark > Update LICENSE/NOTICE for spark-core with shaded dependencie

[jira] [Assigned] (SPARK-48548) Update LICENSE/NOTICE for spark-core with shaded dependencies

[ https://issues.apache.org/jira/browse/SPARK-48548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48548: -- Assignee: (was: Apache Spark) > Update LICENSE/NOTICE for spark-core with shaded

[jira] [Created] (SPARK-48551) Perf improvement for escapePathName

Kent Yao created SPARK-48551: Summary: Perf improvement for escapePathName Key: SPARK-48551 URL: https://issues.apache.org/jira/browse/SPARK-48551 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-48550) Directly use the parent Window class

[ https://issues.apache.org/jira/browse/SPARK-48550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48550: --- Labels: pull-request-available (was: ) > Directly use the parent Window class > ---

[jira] [Created] (SPARK-48550) Directly use the parent Window class

Ruifeng Zheng created SPARK-48550: - Summary: Directly use the parent Window class Key: SPARK-48550 URL: https://issues.apache.org/jira/browse/SPARK-48550 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-48549) Restrict the number of parameters for function `sentences` to 1 or 3

[ https://issues.apache.org/jira/browse/SPARK-48549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48549: --- Labels: pull-request-available (was: ) > Restrict the number of parameters for function `se