holdenk commented on PR #52950: URL: https://github.com/apache/spark/pull/52950#issuecomment-3529525027
Ok did more digging and thinking about the test failures. The problem with the current attempted solution is that we have expression which start out non-deterministic and then become deterministic once a seed assigned but the toAttribute is not updated on that and similarily the unions logic on toAttribute would not be updated on that either. One option I considered was passing in a callback (which I don't love of course) but given that we resolve the attrs in the union once that still would break (although if we made it not cached that could be ok). A "partial" fix could be to only propegate the non-deterministic into attr ref IFF it's something where a seed being set would not resolve the non-determinism but I don't see a super clean way to do that in the code right now. I'm going to give this more thought. Maybe a new trait for "SortOfNonDeterministic" and keep the existing NonDeterministic trait alive? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
