[ 
https://issues.apache.org/jira/browse/HIVE-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509794#comment-14509794
 ] 

Gopal V commented on HIVE-10466:
--------------------------------

The current memory script does this - it uses NODE_MEM/2 as the 
per-instance-executor & uses 1Gb by default as the cache.

Then it goes through a bunch of complex "heuristics" to produce a complete 
configuration listing which contains the YARN container.size, the Xmx and the 
total memory allocated to executors.
 
https://github.com/apache/hive/blob/llap/llap-server/src/main/resources/package.py#L14

This produces a workable configuration, but it misses the total capacity of the 
node by a significant margin (will be >60%, so no double allocs on a single 
node, but will be < 100%).

Even in that script, the yarn min-alloc is missing from the computation, so as 
we edge closer to the line, the harder it gets to configure this correctly.

After that, there's the whole YARN reserved memory fraction to deal with in 
this, so that we can avoid taking up memory in YARN that LLAP can't use.

> LLAP: fix container sizing configuration for memory
> ---------------------------------------------------
>
>                 Key: HIVE-10466
>                 URL: https://issues.apache.org/jira/browse/HIVE-10466
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Gopal V
>            Assignee: Vikram Dixit K
>
> We cannot use full machine for LLAP due to config for cache and executors 
> being "split brain"... please refer to  [~gopalv] for details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to