Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/21212
@squito @cloud-fan @jiangxb1987
Thanks a lot for review.
> shall we also optimize the space usage for MapStatus
@cloud-fan do you mean optimize space usage for MapStatus when there are
lots of consecutive empty-blocks ?
Rashid, thanks a lot for your comments. I refined the doc and did some more
improvement for memory usage. Please take another look.
>curious, have you observed this using a lot of memory from a heapdump or
anything?
Yes, it's from heapdump. We enabled adaptive execution and found this issue.
> I wonder if using an ArrayBuffer is the smartest, maybe it'll have a ton
of wasted space from the way it grows.
I'm not sure about this either. So I just keep the ArrayBuffer here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
