yadavay-amzn commented on PR #55858: URL: https://github.com/apache/spark/pull/55858#issuecomment-4896889984
Thanks for the careful review - you're right, and I incorrectly assumed the V2 MERGE source is materialized before the write. I confirmed it isn't: RewriteMergeIntoTable feeds the source straight into the join (and again into the matching subquery) with no materialization, so allowing non-deterministic source expressions here is unsafe - the two independent source reads (or a task retry) can disagree and produce wrong results. That's exactly SPARK-56683. I'll convert this to a draft and mark it blocked on SPARK-56683's source-materialization work. Happy to review that PR whenever it's up, and thanks again for catching the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
