[jira] [Resolved] (SPARK-48610) Remove ExplainUtils.processPlan synchronize

2024-06-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48610. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-48610][SQL] refactor: use auxiliary idMap instead of OP_ID_TAG

2024-06-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 88582e11519b [SPARK-48610][SQL

(spark) branch master updated: [SPARK-48610][SQL] refactor: use auxiliary idMap instead of OP_ID_TAG

2024-06-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d3da240ee302 [SPARK-48610][SQL] refactor: use

[jira] [Resolved] (SPARK-48557) Support scalar subquery with group-by on column equal to constant

2024-06-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48557. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46902 [https

[jira] [Assigned] (SPARK-48557) Support scalar subquery with group-by on column equal to constant

2024-06-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48557: --- Assignee: Jack Chen > Support scalar subquery with group-by on column equal to const

(spark) branch master updated: [SPARK-48557][SQL] Support scalar subquery with group-by on column equal to constant

2024-06-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 90d302aeb788 [SPARK-48557][SQL] Support scalar

[jira] [Resolved] (SPARK-48577) Replace invalid byte sequences in UTF8Strings

2024-06-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48577. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46899 [https

(spark) branch master updated (0a4b112d87a3 -> 71475f7084b9)

2024-06-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 0a4b112d87a3 [SPARK-48633][BUILD] Upgrade `scalacheck` to 1.18.0 add 71475f7084b9 [SPARK-48577][SQL] Invalid UTF

[jira] [Resolved] (SPARK-48587) Avoid storage amplification when accessing sub-Variant

2024-06-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48587. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46941 [https

(spark) branch master updated: [SPARK-48587][VARIANT] Avoid storage amplification when accessing a sub-Variant

2024-06-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 42cd961fa97d [SPARK-48587][VARIANT] Avoid

[jira] [Resolved] (SPARK-48597) Distinguish the streaming nodes from the text representation of logical plan

2024-06-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48597. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46953 [https

[jira] [Assigned] (SPARK-48597) Distinguish the streaming nodes from the text representation of logical plan

2024-06-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48597: --- Assignee: Jungtaek Lim > Distinguish the streaming nodes from the text representation

(spark) branch master updated: [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan

2024-06-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c09039ad2a95 [SPARK-48597][SQL] Introduce a

(spark) branch master updated: [SPARK-46937][SQL] Revert "[] Improve concurrency performance for FunctionRegistry"

2024-06-11 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 82a84ede6a47 [SPARK-46937][SQL] Revert

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

2024-06-11 Thread Wenchen Fan
at what features are >available in prose. > > > Personally, I think it's fine to caveat what features are available in > prose. For the rare case where we have *completely *incompatible Spark > code (which should be exceedingly rare), we can provide different code > sn

(spark) branch master updated (aad6771aac3d -> 61078366b672)

2024-06-11 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from aad6771aac3d [SPARK-48576][SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE add 61078366b672 [SPARK-48576][SQL

[jira] [Resolved] (SPARK-48576) Rename UTF8_BINARY_LCASE to UTF8_LCASE

2024-06-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48576. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46924 [https

[jira] [Resolved] (SPARK-48556) Incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION

2024-06-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48556. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46900 [https

[jira] [Assigned] (SPARK-48556) Incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION

2024-06-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48556: --- Assignee: Nikola Mandic > Incorrect error message pointing

(spark) branch master updated: [SPARK-48556][SQL] Fix incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION

2024-06-11 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 224ba162b5d6 [SPARK-48556][SQL] Fix incorrect

[jira] [Resolved] (SPARK-47415) Levenshtein (all collations)

2024-06-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47415. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46788 [https

[jira] [Assigned] (SPARK-47415) Levenshtein (all collations)

2024-06-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47415: --- Assignee: Uroš Bojanić > Levenshtein (all collati

(spark) branch master updated: [SPARK-47415][SQL] Add collation support for Levenshtein expression

2024-06-11 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 583ab0500c79 [SPARK-47415][SQL] Add collation

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

2024-06-11 Thread Wenchen Fan
Wed, Jun 5, 2024 at 3:22 PM Neil Ramaswamy >>>> wrote: >>>> >>>>> Thanks all for the responses. Let me try to address everything. >>>>> >>>>> > the programming guides are also different between versions since >>>>>

[jira] [Commented] (SPARK-48562) Writing to JDBC Temporary View Failed

2024-06-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854095#comment-17854095 ] Wenchen Fan commented on SPARK-48562: - writing to a temp view is an ill pattern

[jira] [Assigned] (SPARK-48410) Fix InitCap expression

2024-06-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48410: --- Assignee: Uroš Bojanić > Fix InitCap express

[jira] [Resolved] (SPARK-48410) Fix InitCap expression

2024-06-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48410. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46732 [https

(spark) branch master updated (61fd936c8939 -> 3857a9d36d4a)

2024-06-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 61fd936c8939 [SPARK-48403][SQL] Fix Lower & Upper expressions for UTF8_BINARY_LCASE & ICU collations

[jira] [Resolved] (SPARK-48403) Fix Upper & Lower expressions for UTF8_BINARY_LCASE

2024-06-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48403. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46720 [https

(spark) branch master updated: [SPARK-48403][SQL] Fix Lower & Upper expressions for UTF8_BINARY_LCASE & ICU collations

2024-06-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 61fd936c8939 [SPARK-48403][SQL] Fix Lower

(spark-website) branch asf-site updated: add a behavior change guideline (#518)

2024-06-07 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 93d08b7bff add a behavior change

(spark) branch master updated (d81b1e3d358c -> 8911d59005e8)

2024-06-07 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from d81b1e3d358c [SPARK-48559][SQL] Fetch globalTempDatabase name directly without invoking initialization of

[jira] [Assigned] (SPARK-48559) Fetch globalTempDatabase name directly without invoking initialization of GlobalaTempViewManager

2024-06-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48559: --- Assignee: Wenchen Fan > Fetch globalTempDatabase name directly without invok

[jira] [Resolved] (SPARK-48559) Fetch globalTempDatabase name directly without invoking initialization of GlobalaTempViewManager

2024-06-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48559. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46907 [https

(spark) branch master updated (87b0f5995383 -> d81b1e3d358c)

2024-06-07 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 87b0f5995383 [SPARK-48561][PS][CONNECT] Throw `PandasNotImplementedError` for unsupported plotting functions

[jira] [Assigned] (SPARK-48286) Analyze 'exists' default expression instead of 'current' default expression in structField to v2 column conversion

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48286: --- Assignee: Uros Stankovic > Analyze 'exists' default expression instea

[jira] [Resolved] (SPARK-48286) Analyze 'exists' default expression instead of 'current' default expression in structField to v2 column conversion

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48286. - Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46594 [https

(spark) branch branch-3.5 updated: [SPARK-48286] Fix analysis of column with exists default expression - Add user facing error

2024-06-06 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new a00c11546273 [SPARK-48286] Fix analysis

(spark) branch master updated: [SPARK-48286] Fix analysis of column with exists default expression - Add user facing error

2024-06-06 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0f21df0b29cc [SPARK-48286] Fix analysis of

[jira] [Resolved] (SPARK-48283) Implement modified Lowercase operation for UTF8_BINARY_LCASE

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48283. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46700 [https

(spark) branch master updated: [SPARK-48283][SQL] Modify string comparison for UTF8_BINARY_LCASE

2024-06-06 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 84fa0527834b [SPARK-48283][SQL] Modify string

(spark) branch master updated (3878b57e6e88 -> b5a4b3200362)

2024-06-06 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 3878b57e6e88 [SPARK-48526][SS] Allow passing custom sink to testStream() add b5a4b3200362 [SPARK-48435][SQL

[jira] [Resolved] (SPARK-48435) UNICODE collation should not support binary equality

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48435. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46772 [https

[jira] [Assigned] (SPARK-48435) UNICODE collation should not support binary equality

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48435: --- Assignee: Uroš Bojanić > UNICODE collation should not support binary equal

[jira] [Assigned] (SPARK-48526) Allow passing custom sink to StreamTest::testStream

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48526: --- Assignee: Johan Lasperas > Allow passing custom sink to StreamTest::testStr

[jira] [Resolved] (SPARK-48526) Allow passing custom sink to StreamTest::testStream

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48526. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46866 [https

(spark) branch master updated: [SPARK-48526][SS] Allow passing custom sink to testStream()

2024-06-06 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3878b57e6e88 [SPARK-48526][SS] Allow passing

[jira] [Assigned] (SPARK-48546) Fix ExpressionEncoder after replacing NullPointerExceptions with proper error classes in AssertNotNull expression

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48546: --- Assignee: Daniel > Fix ExpressionEncoder after replacing NullPointerExceptions with pro

[jira] [Resolved] (SPARK-48546) Fix ExpressionEncoder after replacing NullPointerExceptions with proper error classes in AssertNotNull expression

2024-06-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48546. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46888 [https

(spark) branch master updated (7cba1ab4d6ac -> 9f4007f3d89e)

2024-06-06 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 7cba1ab4d6ac [SPARK-48554][INFRA] Use R 4.4.0 in `windows` R GitHub Action Window job add 9f4007f3d89e [SPARK

[jira] [Created] (SPARK-48552) multi-line CSV schema inference should also throw FAILED_READ_FILE

2024-06-06 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48552: --- Summary: multi-line CSV schema inference should also throw FAILED_READ_FILE Key: SPARK-48552 URL: https://issues.apache.org/jira/browse/SPARK-48552 Project: Spark

(spark) branch master updated: [SPARK-47552][CORE][FOLLOWUP] Set spark.hadoop.fs.s3a.connection.establish.timeout to numeric

2024-06-05 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 966c3d9ef1ed [SPARK-47552][CORE][FOLLOWUP] Set

(spark) branch master updated: [SPARK-48307][SQL][FOLLOWUP] Allow outer references in un-referenced CTE relations

2024-06-05 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d5c33c6bfb57 [SPARK-48307][SQL][FOLLOWUP] Allow

(spark) branch master updated (34ac7de89711 -> 490a4b3b1fdf)

2024-06-05 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 34ac7de89711 [SPARK-48536][PYTHON][CONNECT] Cache user specified schema in applyInPandas and applyInArrow add

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

2024-06-05 Thread Wenchen Fan
, but that's probably too much work. Any thoughts? Wenchen On Wed, Jun 5, 2024 at 1:39 AM Martin Andersson wrote: > While I have no practical knowledge of how documentation is maintained in > the spark project, I must agree with Nimrod. For users on older versions, > having a pro

[jira] [Resolved] (SPARK-48307) InlineCTE should keep not-inlined relations in the original WithCTE node

2024-06-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48307. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46617 [https

(spark) branch master updated: [SPARK-48307][SQL] InlineCTE should keep not-inlined relations in the original WithCTE node

2024-06-04 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8a0927c07a14 [SPARK-48307][SQL] InlineCTE

(spark) branch master updated (651f68782ab7 -> c7caac9b10ca)

2024-06-04 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 651f68782ab7 [SPARK-48531][INFRA] Fix `Black` target version to Python 3.9 add c7caac9b10ca [SPARK-47972][SQL

[jira] [Assigned] (SPARK-48318) Hash join support for strings with collation (complex types)

2024-06-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48318: --- Assignee: Uroš Bojanić > Hash join support for strings with collation (complex ty

(spark) branch master updated: [SPARK-48318][SQL] Enable hash join support for all collations (complex types)

2024-06-04 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c852c4f72acb [SPARK-48318][SQL] Enable hash

[jira] [Resolved] (SPARK-48318) Hash join support for strings with collation (complex types)

2024-06-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48318. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46722 [https

(spark) branch master updated: [SPARK-47972][SQL] Restrict CAST expression for collations

2024-06-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e4e8bb5936d3 [SPARK-47972][SQL] Restrict CAST

[jira] [Assigned] (SPARK-47972) Restrict CAST expression for collations

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47972: --- Assignee: Mihailo Milosevic > Restrict CAST expression for collati

[jira] [Resolved] (SPARK-47972) Restrict CAST expression for collations

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47972. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46474 [https

[jira] [Resolved] (SPARK-48413) ALTER COLUMN with collation

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48413. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46734 [https

(spark) branch master updated: [SPARK-48413][SQL] ALTER COLUMN with collation

2024-06-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f9542d008402 [SPARK-48413][SQL] ALTER COLUMN

[ANNOUNCE] Announcing Apache Spark 4.0.0-preview1

2024-06-03 Thread Wenchen Fan
nload Spark 4.0.0-preview1, head over to the download page: https://archive.apache.org/dist/spark/spark-4.0.0-preview1 . It's also available in PyPI, with version name "4.0.0.dev1". Thanks, Wenchen

[ANNOUNCE] Announcing Apache Spark 4.0.0-preview1

2024-06-03 Thread Wenchen Fan
nload Spark 4.0.0-preview1, head over to the download page: https://archive.apache.org/dist/spark/spark-4.0.0-preview1 . It's also available in PyPI, with version name "4.0.0.dev1". Thanks, Wenchen

[jira] [Assigned] (SPARK-48503) Scalar subquery with group-by and non-equality predicate incorrectly allowed, wrong results

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48503: --- Assignee: Jack Chen > Scalar subquery with group-by and non-equality predicate incorrec

[jira] [Resolved] (SPARK-48503) Scalar subquery with group-by and non-equality predicate incorrectly allowed, wrong results

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48503. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46839 [https

(spark) branch master updated: [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed

2024-06-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 5d71ef0716f7 [SPARK-48503][SQL] Fix invalid

svn commit: r69509 - in /dev/spark: v4.0.0-preview1-rc1-bin/ v4.0.0-preview1-rc1-docs/ v4.0.0-preview1-rc2-bin/ v4.0.0-preview1-rc2-docs/

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 01:12:24 2024 New Revision: 69509 Log: Removing RC artifacts. Removed: dev/spark/v4.0.0-preview1-rc1-bin/ dev/spark/v4.0.0-preview1-rc1-docs/ dev/spark/v4.0.0-preview1-rc2-bin/ dev/spark/v4.0.0-preview1-rc2-docs

svn commit: r69508 - /release/spark/KEYS

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 00:59:52 2024 New Revision: 69508 Log: Update KEYS Modified: release/spark/KEYS Modified: release/spark/KEYS == --- release/spark/KEYS (original) +++ release/spark/KEYS Mon Jun 3 00

svn commit: r69507 - /dev/spark/v4.0.0-preview1-rc3-bin/ /release/spark/spark-4.0.0-preview1/

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 00:59:50 2024 New Revision: 69507 Log: Apache Spark 4.0.0-preview1 Added: release/spark/spark-4.0.0-preview1/ - copied from r69506, dev/spark/v4.0.0-preview1-rc3-bin/ Removed: dev/spark/v4.0.0-preview1-rc3-bin

svn commit: r69506 - /dev/spark/v4.0.0-preview1-rc3-docs/

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 00:59:48 2024 New Revision: 69506 Log: Remove RC artifacts Removed: dev/spark/v4.0.0-preview1-rc3-docs/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional

(spark) tag v4.0.0-preview1 created (now 7a7a8bc4bab5)

2024-06-02 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to tag v4.0.0-preview1 in repository https://gitbox.apache.org/repos/asf/spark.git at 7a7a8bc4bab5 (commit) No new revisions were added by this update

Re: [VOTE] SPARK 4.0.0-preview1 (RC3)

2024-06-02 Thread Wenchen Fan
The vote passes with 6+1s (4 binding +1s). (* = binding) +1: Wenchen Fan (*) Kent Yao Cheng Pan Xiao Li (*) Gengliang Wang (*) Tathagata Das (*) Thanks all! On Fri, May 31, 2024 at 6:07 PM Tathagata Das wrote: > +1 > - Tested RC3 with Delta Lake. All our Scala and Python tests pass.

[jira] [Assigned] (SPARK-48391) use addAll instead of add function in TaskMetrics to accelerate

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48391: --- Assignee: jiahong.li > use addAll instead of add function in TaskMetrics to acceler

(spark) branch master updated (96365c86962b -> 3cd35f8cb646)

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 96365c86962b [SPARK-48465][SQL] Avoid no-op empty relation propagation add 3cd35f8cb646 [SPARK-48391][CORE

[jira] [Resolved] (SPARK-48391) use addAll instead of add function in TaskMetrics to accelerate

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48391. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-48391][CORE] Using addAll instead of add function in fromAccumulatorInfos method of TaskMetrics Class

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 744b070fa964 [SPARK-48391][CORE] Using

[jira] [Resolved] (SPARK-48465) Avoid no-op empty relation propagation in AQE

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48465. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46814 [https

(spark) branch master updated (844821c82da5 -> 96365c86962b)

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 844821c82da5 [SPARK-47578][R] Migrate RPackageUtils with variables to structured logging framework add

[jira] [Resolved] (SPARK-48430) Fix map value extraction when map contains collated strings

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48430. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46758 [https

(spark) branch master updated (747437c80aa8 -> f083e61925e9)

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 747437c80aa8 [SPARK-48476][SQL] fix NPE error message for null delmiter csv add f083e61925e9 [SPARK-48430][SQL

[jira] [Resolved] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48476. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46810 [https

[jira] [Assigned] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48476: --- Assignee: Milan Stefanovic > NPE thrown when delimiter set to null in

(spark) branch master updated: [SPARK-48476][SQL] fix NPE error message for null delmiter csv

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 747437c80aa8 [SPARK-48476][SQL] fix NPE error

[jira] [Assigned] (SPARK-48419) Foldable propagation replace foldable column should use origin column

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48419: --- Assignee: KnightChess > Foldable propagation replace foldable column should use ori

[jira] [Resolved] (SPARK-48419) Foldable propagation replace foldable column should use origin column

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48419. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46742 [https

(spark) branch master updated: [SPARK-48419][SQL] Foldable propagation replace foldable column shoul…

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3e27543128c8 [SPARK-48419][SQL] Foldable

[jira] [Resolved] (SPARK-48468) Add LogicalQueryStage interface in catalyst

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48468. - Fix Version/s: 4.0.0 Resolution: Fixed > Add LogicalQueryStage interface in catal

[jira] [Assigned] (SPARK-48468) Add LogicalQueryStage interface in catalyst

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48468: --- Assignee: Ziqi Liu > Add LogicalQueryStage interface in catal

(spark) branch master updated: [SPARK-48468] Add LogicalQueryStage interface in catalyst

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6b4f97e1411c [SPARK-48468] Add

[jira] [Resolved] (SPARK-48477) Refactor CollationSuite, CoalesceShufflePartitionsSuite, SQLExecutionSuite

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48477. - Fix Version/s: 4.0.0 Resolution: Fixed > Refactor CollationSu

(spark) branch master updated: [SPARK-48477][SQL][TESTS] Use withSQLConf in tests: Refactor CollationSuite, CoalesceShufflePartitionsSuite, SQLExecutionSuite

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e1f5a7c856ab [SPARK-48477][SQL][TESTS] Use

[jira] [Resolved] (SPARK-48292) Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48292. - Fix Version/s: 4.0.0 Assignee: angerszhu (was: L. C. Hsieh) Resolution: Fixed

[jira] [Assigned] (SPARK-48292) Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48292: --- Assignee: L. C. Hsieh > Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator sho

(spark) branch master updated (69afd4be9c93 -> f68d761c9b21)

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 69afd4be9c93 [SPARK-47361][SQL] Derby: Calculate suitable precision and scale for DECIMAL type add f68d761c9b21

(spark) branch branch-3.5 updated: [SPARK-41049][SQL][FOLLOW-UP][3.5] stateful expressions test uses different pretty name

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new c87b6483a3e0 [SPARK-41049][SQL][FOLLOW

<    1   2   3   4   5   6   7   8   9   10   >