goutamadwant commented on PR #57809: URL: https://github.com/apache/spark/pull/57809#issuecomment-5521283152
> Btw, the PR description seems a bit overstated? IIUC, there is no production query failure here because the idempotence throw is gated on Utils.isTesting (only true when SPARK_TESTING or spark.testing is set, so basically Spark unit tests only). In production, the batch still runs once and the plan is correct. The leftover reconstruction Project is wasted work, not wrong results, and it does not fail the query. > > I understand this showed up in Delta’s tests because those run in a Spark test JVM, even though Delta’s UPDATE/DELETE uses this plan shape in production too. > > So while this is a reasonable fix, I don't think we should proceed with backports to Spark 4.2 & 4.3, as the ticket suggests. @uros-b thats true. I updated the PR description to clarify the test-only failure and no user-facing correctness impact, and I agree that this should target master only without backports to 4.2 or 4.3. Let me know. thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
