[
https://issues.apache.org/jira/browse/IMPALA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675898#comment-16675898
]
Tim Armstrong commented on IMPALA-691:
--------------------------------------
It would be useful to add this as an option sometime soon - it makes it easier
for people deploying Impala to plan for memory consumption. Currently you need
to manually add the process memory limit and JVM heap together.
> Process mem limit does not account for the JVM's memory usage
> -------------------------------------------------------------
>
> Key: IMPALA-691
> URL: https://issues.apache.org/jira/browse/IMPALA-691
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 1.2.1, Impala 2.0, Impala 2.1, Impala 2.2, Impala
> 2.3.0
> Reporter: Skye Wanderman-Milne
> Assignee: Tim Armstrong
> Priority: Major
> Labels: incompatibility, resource-management
>
> The JVM doesn't appear to use malloc, so it's memory usage is not reported by
> tcmalloc and we do not count it in the process mem limit. I verified this by
> adding a large allocation in the FE, and noting that the total memory usage
> (virtual or resident) reported in /memz is not affected, but the virtual and
> resident memory usage reported by top is.
> This is problematic especially because Impala caches table metadata in the FE
> (JVM) which can become quite big (few GBs) in extreme cases.
> *Workaround*
> As a workaround, we recommend reducing the process memory limit by 1-2GB to
> "reserve" memory for the JVM. How much memory you should reserve typically
> depends on the size of your catalog ( number of
> tables/partitions/columns/blocks etc.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
