Todd Lipcon created IMPALA-7406:
-----------------------------------
Summary: Flatbuffer wrappers use almost as much memory as
underlying data
Key: IMPALA-7406
URL: https://issues.apache.org/jira/browse/IMPALA-7406
Project: IMPALA
Issue Type: Improvement
Components: Catalog
Reporter: Todd Lipcon
Currently the file descriptors stored in the catalogd memory for each partition
use a FlatBuffer to reduce the number of separate objects on the Java heap.
However, the FlatBuffer objects internally each store a ByteBuffer and int
position, so each object takes 32 bytes on its own. The ByteBuffer takes 56
bytes since it stores various references, endianness, limit, mark, position,
etc. This amounts to about 88 bytes overhead on top of the actual underlying
flatbuf byte array which is typically around 100 bytes for a single-block file.
So, we're have about a 1:1 ratio of memory overhead and a 2:1 ratio of object
count overhead for each partition.
If we simply stored the byte[] array and constructed wrappers on demand, we'd
save 88 bytes and 2 objects per partition. The downside is that we'd need to do
short-lived ByteBuffer allocations at access time, and based on some
benchmarking I did, they don't get escape-analyzed out. So, it's not a super
clear win, but still worth considering.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
