Github user squito commented on the issue:
https://github.com/apache/spark/pull/21212
@jinxing64
> I guess your concern is ArrayBuffer will do lots of copy as size of
elements grows, and we don't need fast random access in
ShuffleBlockFetcherIterator
my concern wasn't the copy, my concern was the wasted space at the end of
the ArrayBuffer, as that could waste a lot of space. ArrayBuffers are
initialized with a size of 16:
https://github.com/scala/scala/blob/v2.12.6/src/library/scala/collection/mutable/ArrayBuffer.scala#L63
and they double in size when they grow. Both of those are good in general,
but what if each of those ArrayBuffers only had one item in it? Then you'd
have an array with 15 empty slots. Similarly if you had 17 elements, or 33
elements, etc. I thought the memory inspection should tell you how many
trailing nulls you had in your arrays.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
