-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47506/
-----------------------------------------------------------

(Updated May 18, 2016, 1:57 a.m.)


Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.


Changes
-------

Added Screenshot.


Bugs: AMBARI-16720 and AMBARI-16722
    https://issues.apache.org/jira/browse/AMBARI-16720
    https://issues.apache.org/jira/browse/AMBARI-16722


Repository: ambari


Description
-------

- AMBARI-16720. Update calculation logic for LLAP configs. 
- AMBARI-16722. Change 'Number of LLAP Daemons', 'In-Memory Cache per Daemon', 
'Maximum CPUs per Daemon' to be a 'Text Box' and 'YARN Memory per Daemon' to be 
a 'Label' instead of 'Slider'.


AMBARI-16720 - Following properties can/will get updated based on what 
triggered the calculation.
------------------------------------------------------------------------------------------------

- LLAP concurrency - 
hive-interactive-site/hive.server2.tez.sessions.per.default.queue
- Number of LLAP nodes used - hive-interactive-env/num_llap_nodes
- LLAP YARN container Size (in MB) - 
hive-interactive-site/hive.llap.daemon.yarn.container.mb
- Number of Executors in container - 
hive-interactive-site/hive.llap.daemon.num.executors
- Cache Per node - hive-interactive-site/hive.llap.io.memory.size
- Property related to cache calculation above - 
hive-interactive-site/hive.llap.io.enabled
- LLAP Heap size per node - hive-interactive-env/llap_heap_size
- Slider App Master Container Size - 
hive-interactive-env/slider_am_container_size


The trigger point for updating LLAP configs (mentioned above) is change in 
values of any of the following:

(1). 'enable_hive_interactive' set to 'true' 
(2). 'llap_queue_capacity' 
(3). 'hive.server2.tez.sessions.per.default.queue'
(4). 'llap' named queue getting selected for config 
'hive.llap.daemon.queue.name' and only 2 queues exist ('llap' and 'default') at 
root level.

- If change in value for 'llap_queue_capacity' or 
'hive.server2.tez.sessions.per.default.queue' is detected, that config
value is not calulated, but read and use in calculation for dependent configs.
 
- If at any point of time, calulation cant be done (mostly because dependent 
config can't be retrieved or LLAP queue capacity is not good enough),
the value for configs are set to their minimum or default values. A user can 
retrigger the calulation by sliding the "% of cluster capacity" slider.


Further, as part of 'AMBARI-16722', following configs have been made a Text box 
instead of Slider earlier:

- hive-interactive-env/num_llap_nodes
- hive-interactive-site/hive.llap.io.memory.size
- hive-interactive-site/hive.llap.daemon.num.executors

Following configs has been converted to a Label instead of a Slider earlier:

- hive-interactive-site/hive.llap.daemon.yarn.container.mb


-----------------------------------
Brief on calculations is as follows :
-----------------------------------


LLAP concurrency: 
----------------
Calulated if this is not the driver for calculation, otherwise current value is 
read.

= 25% of LLAP queue size / 'Tez AM container Size' based on YARN min container 
size 


Slider App Master Container Size:
--------------------------------

= Caluclated value lies b/w 256-1024 MB based on it's own value and YARN min 
container size. 


total_am_capacity_required = normalized value of 'tez_am_container_size' based 
on YARN min container size * llap_concurrency + Slider App Master Container Size
cap_available_for_daemons = total_llap_queue_size - total_am_capacity_required
        

Number of LLAP nodes used:
-------------------------

= Normalized value of 'cap_available_for_daemons / yarn_nm_mem_in_mb' based on 
YARN min container size.


LLAP YARN container Size:
------------------------

if Number of LLAP nodes used < 1.00: 
  = Normalized value of 'cap_available_for_daemons' based on YARN min container 
size.
else
  = Normalized value of 'yarn_nm_mem_in_mb' based on YARN min container size.
  

Number of Executors in container:
--------------------------------

= minimum (LLAP YARN container Size / hive_tez_container_size, Number of CPU's 
per Node Manager Host).


total Mem for Executors = Number of Executors in container * Read value of 
hive_tez_container_size


Cache Per node :
--------------

 = LLAP YARN container Size - total Mem for Executors.
 
 hive-interactive-site/hive.llap.io.enabled set to 'false' if 'Cache Per node' 
< 64, else 'true'.
 

LLAP Heap size per node:
-----------------------

 = maximum (total Mem for Executors * 0.8, total Mem for Executors - 1024)


Diffs
-----

  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml
 fffcd03 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
 10518dc 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/themes/theme.json 
90b89f0 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
689e1fd 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py cf82a9c 

Diff: https://reviews.apache.org/r/47506/diff/


Testing
-------

Yes.
- Python UT's added.


File Attachments (updated)
----------------

Screen Shot 2016-05-17 at 6.57.04 PM.png
  
https://reviews.apache.org/media/uploaded/files/2016/05/18/a0b77497-a9f5-47c8-8574-0e5cdc2b66be__Screen_Shot_2016-05-17_at_6.57.04_PM.png


Thanks,

Swapan Shridhar

Reply via email to