ahshahid opened a new pull request, #49708: URL: https://github.com/apache/spark/pull/49708
### What changes were proposed in this pull request? Added a new lazy boolean exprValHasIndeterministicCharacter in the Expression class, to indicate whether the expression's evaluation result was obtained using some indeterministic calculation. The main difference between existing deterministic boolean and this is , that the former determines if same expression is evaluated more than once in a given iteration would its value change or not. Also on that basis, any leaf expression would always be deterministic. But the latter indicates if an expression's evaluation used/uses any non deterministic component. So here even the leaf expression ( like Attribute) can have non deterministic character flag as true, if its pointing to some result obtained via indeterministic calculation. Apart from that code of ShuffleDependency is augmented to indicate if the hashing expression is using any non deterministic component. ### Why are the changes needed? The changes are needed to ensure that in case of non deterministic join keys, the whole Shuffle stage is re-executed instead of just the failing task of the stage. This is achieved by fixing the bug in the Rdd.outputDeterministicLevel which now takes into account the indeterministic nature of ShuffleDependency ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added some basic tests. Will be adding more tests to validate the functional bug of incorrect results. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
