Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/17848
My concern is that people trying non-deterministic UDFs get tripped by
repeated computations at least as often as by internal optimizations, and
`nonDeterministic` flag might send a wrong message.
In particular let's say we have this fan-out - fan-in worfklow depending on
a non-deterministic source:
where dotted edges represent an arbitrary chain of transformations. Can we
ensure that the state of each `foo`descendant in `sinl` will be consistent (`x`
hasn't been recomputed)? I hope my point here is clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
