Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/21736
@MaxGekk you can check that even without my change, ie. with the code of
current master coming from your PR, the result is the same (you can check on
your own). The reason is what was said by @cloud-fan which I try to elaborate a
bit more here.
During analysis/optimization, deterministic operations/functions are
executed and they are replaced in the plan with their result. As the
analysis/optimization is performed only once when the dataframe is created, if
you reuse the same dataframe multiple times you will always get the same
result, as it is stored in the plan. If you want new configs to take effect you
need to create a new dataframe.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
