Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19560
I can see the value and also the potential extra overhead (more expensive
for object stores), although this does not resolve the root cause.
Before we providing adaptive runtime optimization and incremental online
stats collection, this might be a workaround solution for avoiding some OOM
cases. Let us leave this PR open and see the feedbacks from the others
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
