[
https://issues.apache.org/jira/browse/SPARK-57685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091382#comment-18091382
]
Anupam Yadav commented on SPARK-57685:
--------------------------------------
I am working on this as a follow-up to SPARK-56729 (PR #55858).
> Support non-deterministic expressions in MERGE INTO action assignments on
> DSv2 tables
> -------------------------------------------------------------------------------------
>
> Key: SPARK-57685
> URL: https://issues.apache.org/jira/browse/SPARK-57685
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Anupam Yadav
> Priority: Major
>
> Follow-up to SPARK-56729.
> SPARK-56729 (PR #55858) allows non-deterministic expressions in the MERGE
> *source* for DataSource V2 row-level operations (ReplaceData/WriteDelta
> implement SupportsNonDeterministicExpression, with a determinism guard in
> RowLevelOperationRuntimeGroupFiltering to keep runtime group pruning safe).
> However, non-determinism written directly into a MERGE *action* assignment,
> e.g.:
> {code:sql}
> MERGE INTO t USING src ON t.id = src.id
> WHEN MATCHED THEN UPDATE SET t.x = uuid()
> {code}
> still fails analysis with INVALID_NON_DETERMINISTIC_EXPRESSIONS. The
> assignment expression lands on the MergeRows operator, which does not
> implement SupportsNonDeterministicExpression and is not in CheckAnalysis's
> allowlist of operators permitted to carry non-deterministic expressions.
> This is a separate, pre-existing limitation, orthogonal to the
> source-non-determinism regression fixed in SPARK-56729. Supporting it
> requires making MergeRows evaluate/validate such action expressions safely
> (each evaluated once per produced row) or implementing
> SupportsNonDeterministicExpression on MergeRows with execution validation.
> Noted by cloud-fan during review of PR #55858.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
