Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/1880#issuecomment-51737798
I believe this PR can alleviate OOMs a lot. Below are some ideas to make
in-memory columnar store more memory efficient, and can be done in separate PRs
based on this one.
While building column buffers in batch, we still uses 1MB as initial column
buffer size for *each* column (defined as
`ColumnBuilder.DEFAULT_INITIAL_BUFFER_SIZE`). Say T tasks are running in
parallel to squeeze a table with C columns into memory, we allocate at least T
* C * 1MB for each batch.
The initial column buffer size estimation used in Shark can be useful, but
unfortunately the implementation is actually buggy, and usually gives fairly
small initial buffer size. A more reasonable estimation heuristics could be:
1. Let `D[i]` be the default size of the `i`-th column
1. Let `I = sum(D[i]) * batchSize`
1. Default column buffer size for the `i`-th column is `S[i] = I * D[i] /
sum(D[i])`
This estimation is precise for all primitive types whose default sizes
equals to their actual sizes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
