[
https://issues.apache.org/jira/browse/AMBARI-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapan Shridhar updated AMBARI-20225:
-------------------------------------
Description:
**Background :**
**----------**
We are dealing with 2 specific configs here.
**(1).** 'tez.am.resource.memory.mb' for tez-interactive-site and
**(2).** 'hive.tez.container.size' for hive-interactive-site.
- 'tez.am.resource.memory.mb' already exists in tez-interactive-site from
Ambari-2.4 onwards and recently its default value had been changed to
'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
- 'hive.tez.container.size' was inherited and used from Hive1/hive-site in
Ambari 2.4. Starting from Ambari 2.5, it was added as a config explicitly in
hive-interactive-site with a default value of 'SET_ON_FIRST_INVOCATION'.
The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint
in Stack Advisor (SA) call for the 1st time on Hive Server Interactive ON
(enable_hive_interactive config), so that config values for both can be
calculated and set for the 1st time. Subsequesnt calls in the lifecycle, will
just read the value set by SA or if later changed by user manually.
**Problem :**
**---------**
'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade
scenario.
**Fix:**
**----**
Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct
behavior of configs during HDP upgrade.
**Possible scenarios:**
**------------------**
**1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying
2.5/stack_advisor.py code would have been updated with new LLAP GA logic.**
- 'tez.am.resource.memory.mb' will already be there whether Hive Server
Interactive (HSI) is ON or OFF after Ambari upgrade.
- If HSI if OFF before the upgrade was done, nothing to do over there.
- If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade,
SA logic has been updated to take of reading 'hive.tez.container.size' from
hive-site as it wont exist in hive-interactive-site at this point. It will show
up in hive-interative-site aftre the HDP upgrade to 2.6
**2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
- 'tez.am.resource.memory.mb' will already be there
- As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would
have got added, having current value picked from
hive-site/hive.tez.container.size. SA code carries a calculation logic for
config 'hive.tez.container.size', which is bound to yield different value
compared to hive-site/'hive.tez.container.size'. The config's value will get
updated/caluclated (1). when HSI is made ON (meaning 'enabled_hive_interactive'
config detected as **True** in changed-configurations).
**3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
- Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday
present at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
**Note** that a fresh value for 'hive.tez.container.size' and
'tez.am.resource.memory.mb' is caluclated at below conditions:
(1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as
**True** in changed-configurations, or
(2). it's a cluster create operation, where HSI also is supposed to be started.
(BP scenario)
> Take care of Hive2's 'hive.tez.container.size' config during HDP upgrade from
> 2.5 to 2.6
> ----------------------------------------------------------------------------------------
>
> Key: AMBARI-20225
> URL: https://issues.apache.org/jira/browse/AMBARI-20225
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.5.0
> Reporter: Swapan Shridhar
> Assignee: Swapan Shridhar
> Fix For: 2.5.0
>
>
> **Background :**
> **----------**
> We are dealing with 2 specific configs here.
> **(1).** 'tez.am.resource.memory.mb' for tez-interactive-site and
> **(2).** 'hive.tez.container.size' for hive-interactive-site.
> - 'tez.am.resource.memory.mb' already exists in tez-interactive-site from
> Ambari-2.4 onwards and recently its default value had been changed to
> 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
> - 'hive.tez.container.size' was inherited and used from Hive1/hive-site in
> Ambari 2.4. Starting from Ambari 2.5, it was added as a config explicitly in
> hive-interactive-site with a default value of 'SET_ON_FIRST_INVOCATION'.
> The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a
> hint in Stack Advisor (SA) call for the 1st time on Hive Server Interactive
> ON (enable_hive_interactive config), so that config values for both can be
> calculated and set for the 1st time. Subsequesnt calls in the lifecycle, will
> just read the value set by SA or if later changed by user manually.
> **Problem :**
> **---------**
> 'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in
> upgrade scenario.
> **Fix:**
> **----**
> Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct
> behavior of configs during HDP upgrade.
>
> **Possible scenarios:**
> **------------------**
> **1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying
> 2.5/stack_advisor.py code would have been updated with new LLAP GA logic.**
> - 'tez.am.resource.memory.mb' will already be there whether Hive Server
> Interactive (HSI) is ON or OFF after Ambari upgrade.
> - If HSI if OFF before the upgrade was done, nothing to do over there.
> - If HSI on ON before AMbari upgrade, or is made ON after Ambari
> upgrade, SA logic has been updated to take of reading
> 'hive.tez.container.size' from hive-site as it wont exist in
> hive-interactive-site at this point. It will show up in hive-interative-site
> aftre the HDP upgrade to 2.6
> **2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP
> 2.6**
> - 'tez.am.resource.memory.mb' will already be there
> - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would
> have got added, having current value picked from
> hive-site/hive.tez.container.size. SA code carries a calculation logic for
> config 'hive.tez.container.size', which is bound to yield different value
> compared to hive-site/'hive.tez.container.size'. The config's value will get
> updated/caluclated (1). when HSI is made ON (meaning
> 'enabled_hive_interactive' config detected as **True** in
> changed-configurations).
>
> **3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
> - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday
> present at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
> **Note** that a fresh value for 'hive.tez.container.size' and
> 'tez.am.resource.memory.mb' is caluclated at below conditions:
> (1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected
> as **True** in changed-configurations, or
> (2). it's a cluster create operation, where HSI also is supposed to be
> started. (BP scenario)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
