Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20184
>I think that a lazy buffer allocation can not thoroughly solve this
problem because UnsafeSorterSpillReader has BufferedFileInputStream witch will
allocate off heap memory.
Can you please explain more. From my understanding the off heap memory in
`BufferedFileInputStream` is the key issue for your scenario here. I don't
think the logics you changed in `ChainedIterator` matters a lot. So a lazy
allocation of off-heap memory should be enough IIUC.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
