Prasanth Jayachandran created HIVE-17243:
--------------------------------------------
Summary: Replace Hive's DataModel with Java Object Layout
Key: HIVE-17243
URL: https://issues.apache.org/jira/browse/HIVE-17243
Project: Hive
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
org.apache.hadoop.hive.ql.util.JavaDataModel is used for estimating memory of
objects in many places. The problem with this approach is that manual
accounting for all fields and references has to be done. This will also be
problematic for cases where shallow object size vs deep object sizes are
required.
Hash table memory monitoring does accounting for size of the hash tables. The
estimated sizes of hash tables are often very different for non-vector
operators whereas the estimates are close to actual object size for vectorized
operators. Also addition of fields requires manual changes to memory
monitoring.
Java object layout is openjdk project that can provide shallow and deep object
sizes without using java agents. We can leverage that for much more accurate
memory estimates.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
