Todd Lipcon created IMPALA-7501:
-----------------------------------
Summary: Slim down metastore Partition objects in LocalCatalog
cache
Key: IMPALA-7501
URL: https://issues.apache.org/jira/browse/IMPALA-7501
Project: IMPALA
Issue Type: Sub-task
Reporter: Todd Lipcon
I took a heap dump of an impalad running in LocalCatalog mode with a 2G limit
after running a production workload simulation for a couple hours. It had 38.5M
objects and 2.02GB heap (the vast majority of the heap is, as expected, in the
LocalCatalog cache). Of this total footprint, 1.78GB and 34.6M objects are
retained by 'Partition' objects. Drilling into those, 1.29GB and 33.6M objects
are retained by FieldSchema, which, as far as I remember, are ignored on the
partition level by the Impala planner. So, with a bit of slimming down of these
objects, we could make a huge dent in effective cache capacity given a fixed
budget. Reducing object count should also have the effect of improved GC
performance (old gen GC is more closely tied to object count than size)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
