[
https://issues.apache.org/jira/browse/AMBARI-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapan Shridhar updated AMBARI-16720:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Update calculation logic for LLAP configs.
> ------------------------------------------
>
> Key: AMBARI-16720
> URL: https://issues.apache.org/jira/browse/AMBARI-16720
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.4.0
> Reporter: Swapan Shridhar
> Assignee: Swapan Shridhar
> Fix For: 2.4.0
>
> Attachments: AMBARI-16720.patch
>
>
> Following properties can/will get updated based on what triggered the
> calculation.
> - LLAP concurrency -
> hive-interactive-site/hive.server2.tez.sessions.per.default.queue
> - Number of LLAP nodes used - hive-interactive-env/num_llap_nodes
> - LLAP YARN container Size (in MB) -
> hive-interactive-site/hive.llap.daemon.yarn.container.mb
> - Number of Executors in container -
> hive-interactive-site/hive.llap.daemon.num.executors
> - Cache Per node - hive-interactive-site/hive.llap.io.memory.size
> - Property related to cache calculation above -
> hive-interactive-site/hive.llap.io.enabled
> - LLAP Heap size per node - hive-interactive-env/llap_heap_size
> - Slider App Master Container Size -
> hive-interactive-env/slider_am_container_size
> The trigger point for updating LLAP configs (mentioned above) is change in
> values of any of the following:
> (1). 'enable_hive_interactive' set to 'true'
> (2). 'llap_queue_capacity'
> (3). 'hive.server2.tez.sessions.per.default.queue'
> (4). 'llap' named queue getting selected for config
> 'hive.llap.daemon.queue.name' and only 2 queues exist ('llap' and 'default')
> at root level.
> - If change in value for 'llap_queue_capacity' or
> 'hive.server2.tez.sessions.per.default.queue' is detected, that config
> value is not calulated, but read and use in calculation for dependent configs.
> - If at any point of time, calulation cant be done (mostly because dependent
> config can't be retrieved or LLAP queue capacity is not good enough), the
> value for configs are set to their minimum or default values. A user can
> retrigger the calulation by sliding the "% of cluster capacity" slider.
> Brief on calculations is as follows :
> - LLAP concurrency:
> {code}
> Calculated if this is not the driver for calculation, otherwise current value
> is read.
> = 25% of LLAP queue size / 'Tez AM container Size' based on YARN min
> container size
> {code}
> - Slider App Master Container Size:
> {code}
> = Calcuclated value lies b/w 256-1024 MB based on it's own value and YARN min
> container size.
> total_am_capacity_required = normalized value of 'tez_am_container_size'
> based on YARN min container size * llap_concurrency + Slider App Master
> Container Size
> cap_available_for_daemons = total_llap_queue_size - total_am_capacity_required
> {code}
> - Number of LLAP nodes used:
> {code}
> = Normalized value of 'cap_available_for_daemons / yarn_nm_mem_in_mb' based
> on YARN min container size.
> {code}
> - LLAP YARN container Size:
> {code}
> if Number of LLAP nodes used < 1.00:
> = Normalized value of 'cap_available_for_daemons' based on YARN min
> container size.
> else
> = Normalized value of 'yarn_nm_mem_in_mb' based on YARN min container size.
> {code}
> - Number of Executors in container:
> {code}
> = minimum (LLAP YARN container Size / hive_tez_container_size, Number of
> CPU's per Node Manager Host).
> total Mem for Executors = Number of Executors in container * Read value of
> hive_tez_container_size
> {code}
> - Cache Per node :
> {code}
> = LLAP YARN container Size - total Mem for Executors.
> hive-interactive-site/hive.llap.io.enabled set to 'false' if 'Cache Per node'
> < 64, else 'true'.
> {code}
> - LLAP Heap size per node:
> {code}
> = maximum (total Mem for Executors * 0.8, total Mem for Executors - 1024)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
