Github user juliuszsompolski commented on the issue:
https://github.com/apache/spark/pull/19181
LGTM.
It'd be nicer if there was an invariant that a nested spill just never can
happen, i.e. there are no places that can spill on the code path that already
spills.
But this fix does seem to render nested spill harmless in this particular
case.
@hvanhovell In my original use case I hit this bug after hours of running
on a big cluster, so reproducing it on that case is quite unpractical for me.
I'll see if I'll observe it again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
