[
https://issues.apache.org/jira/browse/IMPALA-7501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629410#comment-16629410
]
Paul Rogers edited comment on IMPALA-7501 at 9/26/18 11:35 PM:
---------------------------------------------------------------
A quick scan of the Hive code suggests that Hive's Thrift objects carry more
info that is required in the Impala cache. Creating Impala-specific,
high-performance versions would likely save space. (No need for parent
pointers, no need for the two-level Hive API structure, etc.)
So, this gives us two options:
* Reach inside Hive's Thrift objects to null out fields which we don't need, or
* Design an Impala-specific, compact representation for the data that omits all
but essential objects and fields.
The second choice provides a huge opportunity for memory optimization. The
first is a crude-but-effective short-term solution.
But, see a later note, the story is not as simple as a first analysis suggests.
was (Author: paul.rogers):
Analysis:
* Impala's {{LocalCatalog}} contains a list of {{FeDb}} objects.
* Impala's {{LocalDb}}, which extends {{FeDb}} contains a map of {{LocalTable}}
objects.
* Impala's {{LocalTable}} contains a Hive {{Table}} object.
* The {{Table}} object is defined in [Hive's Thrift
schema|https://github.com/apache/hive/blob/3287a097e31063cc805ca55c2ca7defffe761b6f/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift]
API. It does not contain a list of partitions.
* The {{LocalTable}} wraps a number of subclass, of which the one of interest
is {{HdfsTable}}.
Impala loads tables in the background by calling {{HdfsTable.load()}}:
* {{load()}} calls {{loadAllPartitions()}} to do the partition work.
* {{loadAllPartitions}} calls {{MetaStoreUtil.fetchAllPartitions()}} to get the
partitions as a list of Hive {{Partition} objects.
* {{loadAllParitions}} wraps each in a {{HdfsPartition}}, and calls
{{addPartition}} to put the partition into a couple of maps.
* Hive's
[{{Partition}}|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java]
is generated from Thrift. Contains a {{StorageDescriptor}}.
* Hive's
[{{StorageDescriptor}}|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java]
contains the list of {{FieldSchema}} objects which Todd saw in the heap dump.
Things are a bit confusing because:
* Hive defines a different
[{{Table}}|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java]
class, which contains a {{TableSpec}}.
* Hive's
[{{TableSpec}}|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java]
contains a list of {{Partition}} objects.
A quick scan of the Hive code suggests that Hive's Thrift objects carry more
info that is required in the Impala cache. Creating Impala-specific,
high-performance versions would likely save space. (No need for parent
pointers, no need for the two-level Hive API structure, etc.)
So, this gives us two options:
* Reach inside Hive's Thrift objects to null out fields which we don't need, or
* Design an Impala-specific, compact representation for the data that omits all
but essential objects and fields.
The second choice provides a huge opportunity for memory optimization. The
first is a crude-but-effective short-term solution.
> Slim down metastore Partition objects in LocalCatalog cache
> -----------------------------------------------------------
>
> Key: IMPALA-7501
> URL: https://issues.apache.org/jira/browse/IMPALA-7501
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Todd Lipcon
> Priority: Minor
>
> I took a heap dump of an impalad running in LocalCatalog mode with a 2G limit
> after running a production workload simulation for a couple hours. It had
> 38.5M objects and 2.02GB heap (the vast majority of the heap is, as expected,
> in the LocalCatalog cache). Of this total footprint, 1.78GB and 34.6M objects
> are retained by 'Partition' objects. Drilling into those, 1.29GB and 33.6M
> objects are retained by FieldSchema, which, as far as I remember, are ignored
> on the partition level by the Impala planner. So, with a bit of slimming down
> of these objects, we could make a huge dent in effective cache capacity given
> a fixed budget. Reducing object count should also have the effect of improved
> GC performance (old gen GC is more closely tied to object count than size)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
