[jira] [Resolved] (SPARK-43533) Enable MultiIndex test for IndexesTests.test_difference

[ https://issues.apache.org/jira/browse/SPARK-43533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43533. - Fix Version/s: 4.0.0 Target Version/s: 4.0.0 Resolution: Fixed > Enable

[jira] [Resolved] (SPARK-43499) Enable StatsTests.test_stat_functions_with_no_numeric_columns for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43499. - Fix Version/s: 4.0.0 Target Version/s: 4.0.0 Resolution: Fixed This is

[jira] [Resolved] (SPARK-43497) Enable StatsTests.test_cov_corr_meta for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43497. - Fix Version/s: 4.0.0 Target Version/s: 4.0.0 Resolution: Fixed This is

[jira] [Created] (SPARK-45076) Switch to built-in repeat function

Ruifeng Zheng created SPARK-45076: - Summary: Switch to built-in repeat function Key: SPARK-45076 URL: https://issues.apache.org/jira/browse/SPARK-45076 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-45075) Alter table with invalid default value will not report error

[ https://issues.apache.org/jira/browse/SPARK-45075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-45075: Summary: Alter table with invalid default value will not report error (was: Alter table with invaild

[jira] [Created] (SPARK-45075) Alter table with invaild default value will not report error

Jia Fan created SPARK-45075: --- Summary: Alter table with invaild default value will not report error Key: SPARK-45075 URL: https://issues.apache.org/jira/browse/SPARK-45075 Project: Spark Issue

[jira] [Resolved] (SPARK-43241) MultiIndex.append not checking names for equality

[ https://issues.apache.org/jira/browse/SPARK-43241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43241. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42787

[jira] [Assigned] (SPARK-43241) MultiIndex.append not checking names for equality

[ https://issues.apache.org/jira/browse/SPARK-43241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43241: - Assignee: Haejoon Lee > MultiIndex.append not checking names for equality >

[jira] [Assigned] (SPARK-45073) Replace LastNotNull with Last(ignoreNulls=True)

[ https://issues.apache.org/jira/browse/SPARK-45073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45073: - Assignee: Ruifeng Zheng > Replace LastNotNull with Last(ignoreNulls=True) >

[jira] [Resolved] (SPARK-45073) Replace LastNotNull with Last(ignoreNulls=True)

[ https://issues.apache.org/jira/browse/SPARK-45073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45073. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42808

[jira] [Created] (SPARK-45074) DataFrame.{sort, sortWithinPartitions} support column ordinals

Ruifeng Zheng created SPARK-45074: - Summary: DataFrame.{sort, sortWithinPartitions} support column ordinals Key: SPARK-45074 URL: https://issues.apache.org/jira/browse/SPARK-45074 Project: Spark

[jira] [Resolved] (SPARK-45049) Refine docstrings of `coalesce/repartition/repartitionByRange`

[ https://issues.apache.org/jira/browse/SPARK-45049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45049. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42770

[jira] [Assigned] (SPARK-45049) Refine docstrings of `coalesce/repartition/repartitionByRange`

[ https://issues.apache.org/jira/browse/SPARK-45049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45049: Assignee: Ruifeng Zheng > Refine docstrings of `coalesce/repartition/repartitionByRange`

[jira] [Resolved] (SPARK-42619) Add `show_counts` parameter for DataFrame.info

[ https://issues.apache.org/jira/browse/SPARK-42619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-42619. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 40436

[jira] [Created] (SPARK-45073) Replace LastNotNull with Last(ignoreNulls=True)

Ruifeng Zheng created SPARK-45073: - Summary: Replace LastNotNull with Last(ignoreNulls=True) Key: SPARK-45073 URL: https://issues.apache.org/jira/browse/SPARK-45073 Project: Spark Issue

[jira] [Updated] (SPARK-45073) Replace LastNotNull with Last(ignoreNulls=True)

[ https://issues.apache.org/jira/browse/SPARK-45073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-45073: -- Issue Type: Improvement (was: New Feature) > Replace LastNotNull with Last(ignoreNulls=True)

[jira] [Updated] (SPARK-44805) Data lost after union using spark.sql.parquet.enableNestedColumnVectorizedReader=true

[ https://issues.apache.org/jira/browse/SPARK-44805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-44805: -- Labels: correctness (was: ) > Data lost after union using >

[jira] [Assigned] (SPARK-45064) Add the scale parameter in ceil/ceiling

[ https://issues.apache.org/jira/browse/SPARK-45064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45064: - Assignee: Ruifeng Zheng > Add the scale parameter in ceil/ceiling >

[jira] [Resolved] (SPARK-45064) Add the scale parameter in ceil/ceiling

[ https://issues.apache.org/jira/browse/SPARK-45064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45064. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42791

[jira] [Assigned] (SPARK-45047) DataFrame.groupBy support Ordinal input

[ https://issues.apache.org/jira/browse/SPARK-45047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45047: - Assignee: Ruifeng Zheng > DataFrame.groupBy support Ordinal input >

[jira] [Resolved] (SPARK-45047) DataFrame.groupBy support Ordinal input

[ https://issues.apache.org/jira/browse/SPARK-45047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45047. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42767

[jira] [Assigned] (SPARK-45066) Make function `repeat` accept column-type `n`

[ https://issues.apache.org/jira/browse/SPARK-45066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45066: - Assignee: Ruifeng Zheng > Make function `repeat` accept column-type `n` >

[jira] [Resolved] (SPARK-45066) Make function `repeat` accept column-type `n`

[ https://issues.apache.org/jira/browse/SPARK-45066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45066. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42794

[jira] [Assigned] (SPARK-45063) Refine docstrings of `min_by/max_by`

[ https://issues.apache.org/jira/browse/SPARK-45063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45063: - Assignee: Yang Jie > Refine docstrings of `min_by/max_by` >

[jira] [Resolved] (SPARK-45063) Refine docstrings of `min_by/max_by`

[ https://issues.apache.org/jira/browse/SPARK-45063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45063. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42789

[jira] [Assigned] (SPARK-45031) Choose the right merge code path and merge hash for reopened PRs

[ https://issues.apache.org/jira/browse/SPARK-45031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45031: - Assignee: Kent Yao > Choose the right merge code path and merge hash for reopened PRs

[jira] [Resolved] (SPARK-45031) Choose the right merge code path and merge hash for reopened PRs

[ https://issues.apache.org/jira/browse/SPARK-45031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45031. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42749

[jira] [Commented] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761892#comment-17761892 ] Dongjoon Hyun commented on SPARK-44940: --- This is backported to branch-3.4 via

[jira] [Updated] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44940: -- Fix Version/s: 3.4.2 3.5.1 (was: 3.5.0) > Improve

[jira] [Resolved] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44940. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42790

[jira] [Assigned] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44940: - Assignee: Ivan Sadikov > Improve performance of JSON parsing when >

[jira] [Updated] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44940: -- Issue Type: Bug (was: Improvement) > Improve performance of JSON parsing when >

[jira] [Updated] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44940: -- Labels: correctness (was: ) > Improve performance of JSON parsing when >

[jira] [Resolved] (SPARK-45036) SPJ: Refactor logic to handle partially clustered distribution

[ https://issues.apache.org/jira/browse/SPARK-45036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45036. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42757

[jira] [Assigned] (SPARK-45036) SPJ: Refactor logic to handle partially clustered distribution

[ https://issues.apache.org/jira/browse/SPARK-45036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45036: - Assignee: Chao Sun > SPJ: Refactor logic to handle partially clustered distribution >

[jira] [Created] (SPARK-45072) Fix Outerscopes for same cell evaluation

Herman van Hövell created SPARK-45072: - Summary: Fix Outerscopes for same cell evaluation Key: SPARK-45072 URL: https://issues.apache.org/jira/browse/SPARK-45072 Project: Spark Issue

[jira] [Assigned] (SPARK-45067) Upgrade slf4j to 2.0.9

[ https://issues.apache.org/jira/browse/SPARK-45067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-45067: Assignee: Yang Jie > Upgrade slf4j to 2.0.9 > -- > >

[jira] [Resolved] (SPARK-45067) Upgrade slf4j to 2.0.9

[ https://issues.apache.org/jira/browse/SPARK-45067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45067. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42796

[jira] [Updated] (SPARK-45067) Upgrade slf4j to 2.0.9

[ https://issues.apache.org/jira/browse/SPARK-45067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45067: - Priority: Trivial (was: Major) > Upgrade slf4j to 2.0.9 > -- > >

[jira] [Resolved] (SPARK-44890) Miswritten remarks in pom file

[ https://issues.apache.org/jira/browse/SPARK-44890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44890. -- Fix Version/s: 4.0.0 Assignee: chenyu Resolution: Fixed Resolved by

[jira] [Updated] (SPARK-44890) Miswritten remarks in pom file

[ https://issues.apache.org/jira/browse/SPARK-44890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-44890: - Issue Type: Improvement (was: Bug) This is too trivial for a JIRA > Miswritten remarks in pom

[jira] [Updated] (SPARK-45042) Upgrade jetty to 9.4.52.v20230823

[ https://issues.apache.org/jira/browse/SPARK-45042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-45042: - Fix Version/s: 3.5.0 > Upgrade jetty to 9.4.52.v20230823 > - >

[jira] [Updated] (SPARK-45071) Optimize the processing speed of `BinaryArithmetic#dataType` when processing multi-column data

[ https://issues.apache.org/jira/browse/SPARK-45071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming95 updated SPARK-45071: --- Description: Since `BinaryArithmetic#dataType` will recursively process the datatype of each node,

[jira] [Created] (SPARK-45071) Optimize the processing speed of `BinaryArithmetic#dataType` when processing multi-column data

ming95 created SPARK-45071: -- Summary: Optimize the processing speed of `BinaryArithmetic#dataType` when processing multi-column data Key: SPARK-45071 URL: https://issues.apache.org/jira/browse/SPARK-45071

[jira] [Updated] (SPARK-45071) Optimize the processing speed of `BinaryArithmetic#dataType` when processing multi-column data

[ https://issues.apache.org/jira/browse/SPARK-45071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming95 updated SPARK-45071: --- Issue Type: Improvement (was: Bug) > Optimize the processing speed of `BinaryArithmetic#dataType`

[jira] [Created] (SPARK-45070) Describe the binary and datetime formats of `to_char`/`to_varchar`

Max Gekk created SPARK-45070: Summary: Describe the binary and datetime formats of `to_char`/`to_varchar` Key: SPARK-45070 URL: https://issues.apache.org/jira/browse/SPARK-45070 Project: Spark

[jira] [Updated] (SPARK-38101) MetadataFetchFailedException due to decommission block migrations

[ https://issues.apache.org/jira/browse/SPARK-38101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emil Ejbyfeldt updated SPARK-38101: --- Description: As noted in SPARK-34939 there is race when using broadcast for map output

[jira] [Updated] (SPARK-45020) org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'default' not found (state=08S01,code=0)

[ https://issues.apache.org/jira/browse/SPARK-45020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-45020: Fix Version/s: (was: 3.1.0) > org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException:

[jira] [Assigned] (SPARK-44846) PushFoldableIntoBranches in complex grouping expressions may cause bindReference error

[ https://issues.apache.org/jira/browse/SPARK-44846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-44846: --- Assignee: zhuml > PushFoldableIntoBranches in complex grouping expressions may cause >

[jira] [Resolved] (SPARK-44846) PushFoldableIntoBranches in complex grouping expressions may cause bindReference error

[ https://issues.apache.org/jira/browse/SPARK-44846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-44846. - Fix Version/s: 3.5.0 4.0.0 3.4.2 Resolution: Fixed

[jira] [Created] (SPARK-45069) SQL variable should always be resolved after outer reference

Wenchen Fan created SPARK-45069: --- Summary: SQL variable should always be resolved after outer reference Key: SPARK-45069 URL: https://issues.apache.org/jira/browse/SPARK-45069 Project: Spark

[jira] [Comment Edited] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761796#comment-17761796 ] Paul Praet edited comment on SPARK-36203 at 9/4/23 11:55 AM: - This problem

[jira] [Commented] (SPARK-36203) Spark SQL can't use "group by" on the column of map type.

[ https://issues.apache.org/jira/browse/SPARK-36203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761796#comment-17761796 ] Paul Praet commented on SPARK-36203: This problem is still present in 3.3.2 (and I think also in

[jira] [Resolved] (SPARK-45033) Support maps constructed from arrays in parameterized sql()

[ https://issues.apache.org/jira/browse/SPARK-45033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-45033. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42752

[jira] [Resolved] (SPARK-45059) Add try_reflect to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-45059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-45059. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42783

[jira] [Assigned] (SPARK-45059) Add try_reflect to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-45059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-45059: - Assignee: Jia Fan > Add try_reflect to Scala and Python >

[jira] [Resolved] (SPARK-45060) to_char() fails with an internal error on NULL format

[ https://issues.apache.org/jira/browse/SPARK-45060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-45060. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42781

[jira] [Created] (SPARK-45068) Make function output column name consistent in case

BingKun Pan created SPARK-45068: --- Summary: Make function output column name consistent in case Key: SPARK-45068 URL: https://issues.apache.org/jira/browse/SPARK-45068 Project: Spark Issue

[jira] [Created] (SPARK-45067) Upgrade slf4j to 2.0.9

Yang Jie created SPARK-45067: Summary: Upgrade slf4j to 2.0.9 Key: SPARK-45067 URL: https://issues.apache.org/jira/browse/SPARK-45067 Project: Spark Issue Type: Improvement Components: