Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/15713
+CC @rxin who can elaborate better.
The reason iirc why the first two variables are inline is to do with usual
low cardinality of the buffer - it effectively comes for "free" due to padding
overheads of an object : lowering number of objects created and so gc impact.
For specific workloads, ofcourse this might not be the case.
Having said that, this was done in 1.0-1.1 timeframe, and perhaps not
applicable anymore ?
A more direct optimization could be to have specialized versions of the
class for the primitive types (or rely on something like trove or apache
collections). We used to do that for some pretty remarkable when working with
primitives - for both AppendOnlyMap and CompactBuffer
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
