-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47506/#review133692
-----------------------------------------------------------


Ship it!




We have gone through the code change in person.

- Sumit Mohanty


On May 18, 2016, 1:57 a.m., Swapan Shridhar wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47506/
> -----------------------------------------------------------
> 
> (Updated May 18, 2016, 1:57 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-16720 and AMBARI-16722
>     https://issues.apache.org/jira/browse/AMBARI-16720
>     https://issues.apache.org/jira/browse/AMBARI-16722
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> - AMBARI-16720. Update calculation logic for LLAP configs. 
> - AMBARI-16722. Change 'Number of LLAP Daemons', 'In-Memory Cache per 
> Daemon', 'Maximum CPUs per Daemon' to be a 'Text Box' and 'YARN Memory per 
> Daemon' to be a 'Label' instead of 'Slider'.
> 
> 
> AMBARI-16720 - Following properties can/will get updated based on what 
> triggered the calculation.
> ------------------------------------------------------------------------------------------------
> 
> - LLAP concurrency - 
> hive-interactive-site/hive.server2.tez.sessions.per.default.queue
> - Number of LLAP nodes used - hive-interactive-env/num_llap_nodes
> - LLAP YARN container Size (in MB) - 
> hive-interactive-site/hive.llap.daemon.yarn.container.mb
> - Number of Executors in container - 
> hive-interactive-site/hive.llap.daemon.num.executors
> - Cache Per node - hive-interactive-site/hive.llap.io.memory.size
> - Property related to cache calculation above - 
> hive-interactive-site/hive.llap.io.enabled
> - LLAP Heap size per node - hive-interactive-env/llap_heap_size
> - Slider App Master Container Size - 
> hive-interactive-env/slider_am_container_size
> 
> 
> The trigger point for updating LLAP configs (mentioned above) is change in 
> values of any of the following:
> 
> (1). 'enable_hive_interactive' set to 'true' 
> (2). 'llap_queue_capacity' 
> (3). 'hive.server2.tez.sessions.per.default.queue'
> (4). 'llap' named queue getting selected for config 
> 'hive.llap.daemon.queue.name' and only 2 queues exist ('llap' and 'default') 
> at root level.
> 
> - If change in value for 'llap_queue_capacity' or 
> 'hive.server2.tez.sessions.per.default.queue' is detected, that config
> value is not calulated, but read and use in calculation for dependent configs.
>  
> - If at any point of time, calulation cant be done (mostly because dependent 
> config can't be retrieved or LLAP queue capacity is not good enough),
> the value for configs are set to their minimum or default values. A user can 
> retrigger the calulation by sliding the "% of cluster capacity" slider.
> 
> 
> Further, as part of 'AMBARI-16722', following configs have been made a Text 
> box instead of Slider earlier:
> 
> - hive-interactive-env/num_llap_nodes
> - hive-interactive-site/hive.llap.io.memory.size
> - hive-interactive-site/hive.llap.daemon.num.executors
> 
> Following configs has been converted to a Label instead of a Slider earlier:
> 
> - hive-interactive-site/hive.llap.daemon.yarn.container.mb
> 
> 
> -----------------------------------
> Brief on calculations is as follows :
> -----------------------------------
> 
> 
> LLAP concurrency: 
> ----------------
> Calulated if this is not the driver for calculation, otherwise current value 
> is read.
> 
> = 25% of LLAP queue size / 'Tez AM container Size' based on YARN min 
> container size 
> 
> 
> Slider App Master Container Size:
> --------------------------------
> 
> = Caluclated value lies b/w 256-1024 MB based on it's own value and YARN min 
> container size. 
> 
> 
> total_am_capacity_required = normalized value of 'tez_am_container_size' 
> based on YARN min container size * llap_concurrency + Slider App Master 
> Container Size
> cap_available_for_daemons = total_llap_queue_size - total_am_capacity_required
>         
> 
> Number of LLAP nodes used:
> -------------------------
> 
> = Normalized value of 'cap_available_for_daemons / yarn_nm_mem_in_mb' based 
> on YARN min container size.
> 
> 
> LLAP YARN container Size:
> ------------------------
> 
> if Number of LLAP nodes used < 1.00: 
>   = Normalized value of 'cap_available_for_daemons' based on YARN min 
> container size.
> else
>   = Normalized value of 'yarn_nm_mem_in_mb' based on YARN min container size.
>   
> 
> Number of Executors in container:
> --------------------------------
> 
> = minimum (LLAP YARN container Size / hive_tez_container_size, Number of 
> CPU's per Node Manager Host).
> 
> 
> total Mem for Executors = Number of Executors in container * Read value of 
> hive_tez_container_size
> 
> 
> Cache Per node :
> --------------
> 
>  = LLAP YARN container Size - total Mem for Executors.
>  
>  hive-interactive-site/hive.llap.io.enabled set to 'false' if 'Cache Per 
> node' < 64, else 'true'.
>  
> 
> LLAP Heap size per node:
> -----------------------
> 
>  = maximum (total Mem for Executors * 0.8, total Mem for Executors - 1024)
> 
> 
> Diffs
> -----
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml
>  fffcd03 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
>  10518dc 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/themes/theme.json
>  90b89f0 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 689e1fd 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> cf82a9c 
> 
> Diff: https://reviews.apache.org/r/47506/diff/
> 
> 
> Testing
> -------
> 
> Yes.
> - Python UT's added.
> 
> 
> File Attachments
> ----------------
> 
> Screen Shot 2016-05-17 at 6.57.04 PM.png
>   
> https://reviews.apache.org/media/uploaded/files/2016/05/18/a0b77497-a9f5-47c8-8574-0e5cdc2b66be__Screen_Shot_2016-05-17_at_6.57.04_PM.png
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Reply via email to