Re: Review Request 50409: Adding Services After A Stack/Ambari Upgrade Shows Empty Required Values

2016-07-26 Thread Dmitro Lisnichenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50409/#review143548
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On July 26, 2016, 6:52 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50409/
> ---
> 
> (Updated July 26, 2016, 6:52 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-17894
> https://issues.apache.org/jira/browse/AMBARI-17894
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> 
> - Install Ambari 2.2.2 with Hive on HDP 2.4.2.0 and Kerberize
> - Upgrade to Ambari 2.4.0
> - Upgrade to HDP 2.5
> - Distribute Keytabs
> - Add a new service
> 
> At this point, the UI flags Hive as having configurations which need 
> attention. The follow are all blank and are marked as required:
> 
> {code}
> hive-site/hive.server2.authentication.ldap.url
> hiveserver2-site/hive.conf.restricted.list
> hiveserver2-site/hive.security.authenticator.manager
> hiveserver2-site/hive.security.authorization.manager
> {code}
> 
> There are actually two problems here:
> 
> - The Kerberos wizard interprets the stack advisor "delete" attribute and 
> improperly sets config properties to blank instead of actually removing them
> 
> - The upgrade logic is adding properties back when it should not.
> 
> The Kerberos issue aside, we can't be adding properties back during upgrade 
> stack merging when those properties were specifically removed by the stack 
> advisor prior.
> 
> For example (Ambari 2.2.2 installed with HDP 2.x)
> - Ambari 2.2.2 does not have {{foo-site/property}} for HDP 2.x
> - Ambari 2.4.0 adds {{foo-site/property}} for HDP 2.x, but instructs the 
> upgrade not to add it
> - An upgrade to HDP 2.y sees that {{foo-site/property}} doesn't exist and 
> thinks it's brand new and needs to be merged.
> 
> The upgrade logic should check to see if {{foo-site/property}} is part of 
> both 2.x and 2.y default configurations. If it's part of both of them and is 
> not currently set, then upgrade should NOT set it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  55e9dbd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hiveserver2-site.xml
>  ad37114 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackUpgradeConfigurationMergeTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50409/diff/
> 
> 
> Testing
> ---
> 
> Added unit tests
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>



Re: Review Request 50409: Adding Services After A Stack/Ambari Upgrade Shows Empty Required Values

2016-07-26 Thread Jonathan Hurley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50409/
---

(Updated July 26, 2016, 11:52 a.m.)


Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.


Bugs: AMBARI-17894
https://issues.apache.org/jira/browse/AMBARI-17894


Repository: ambari


Description
---

STR:

- Install Ambari 2.2.2 with Hive on HDP 2.4.2.0 and Kerberize
- Upgrade to Ambari 2.4.0
- Upgrade to HDP 2.5
- Distribute Keytabs
- Add a new service

At this point, the UI flags Hive as having configurations which need attention. 
The follow are all blank and are marked as required:

{code}
hive-site/hive.server2.authentication.ldap.url
hiveserver2-site/hive.conf.restricted.list
hiveserver2-site/hive.security.authenticator.manager
hiveserver2-site/hive.security.authorization.manager
{code}

There are actually two problems here:

- The Kerberos wizard interprets the stack advisor "delete" attribute and 
improperly sets config properties to blank instead of actually removing them

- The upgrade logic is adding properties back when it should not.

The Kerberos issue aside, we can't be adding properties back during upgrade 
stack merging when those properties were specifically removed by the stack 
advisor prior.

For example (Ambari 2.2.2 installed with HDP 2.x)
- Ambari 2.2.2 does not have {{foo-site/property}} for HDP 2.x
- Ambari 2.4.0 adds {{foo-site/property}} for HDP 2.x, but instructs the 
upgrade not to add it
- An upgrade to HDP 2.y sees that {{foo-site/property}} doesn't exist and 
thinks it's brand new and needs to be merged.

The upgrade logic should check to see if {{foo-site/property}} is part of both 
2.x and 2.y default configurations. If it's part of both of them and is not 
currently set, then upgrade should NOT set it.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 55e9dbd 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hiveserver2-site.xml
 ad37114 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackUpgradeConfigurationMergeTest.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/50409/diff/


Testing (updated)
---

Added unit tests


Thanks,

Jonathan Hurley



Re: Review Request 50409: Adding Services After A Stack/Ambari Upgrade Shows Empty Required Values

2016-07-26 Thread Nate Cole

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50409/#review143515
---


Ship it!




Ship It!

- Nate Cole


On July 25, 2016, 5:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50409/
> ---
> 
> (Updated July 25, 2016, 5:04 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-17894
> https://issues.apache.org/jira/browse/AMBARI-17894
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> 
> - Install Ambari 2.2.2 with Hive on HDP 2.4.2.0 and Kerberize
> - Upgrade to Ambari 2.4.0
> - Upgrade to HDP 2.5
> - Distribute Keytabs
> - Add a new service
> 
> At this point, the UI flags Hive as having configurations which need 
> attention. The follow are all blank and are marked as required:
> 
> {code}
> hive-site/hive.server2.authentication.ldap.url
> hiveserver2-site/hive.conf.restricted.list
> hiveserver2-site/hive.security.authenticator.manager
> hiveserver2-site/hive.security.authorization.manager
> {code}
> 
> There are actually two problems here:
> 
> - The Kerberos wizard interprets the stack advisor "delete" attribute and 
> improperly sets config properties to blank instead of actually removing them
> 
> - The upgrade logic is adding properties back when it should not.
> 
> The Kerberos issue aside, we can't be adding properties back during upgrade 
> stack merging when those properties were specifically removed by the stack 
> advisor prior.
> 
> For example (Ambari 2.2.2 installed with HDP 2.x)
> - Ambari 2.2.2 does not have {{foo-site/property}} for HDP 2.x
> - Ambari 2.4.0 adds {{foo-site/property}} for HDP 2.x, but instructs the 
> upgrade not to add it
> - An upgrade to HDP 2.y sees that {{foo-site/property}} doesn't exist and 
> thinks it's brand new and needs to be merged.
> 
> The upgrade logic should check to see if {{foo-site/property}} is part of 
> both 2.x and 2.y default configurations. If it's part of both of them and is 
> not currently set, then upgrade should NOT set it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  55e9dbd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hiveserver2-site.xml
>  ad37114 
> 
> Diff: https://reviews.apache.org/r/50409/diff/
> 
> 
> Testing
> ---
> 
> PENDING UNIT TESTS
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>



Re: Review Request 50409: Adding Services After A Stack/Ambari Upgrade Shows Empty Required Values

2016-07-25 Thread Jonathan Hurley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50409/#review143427
---




ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 (lines 1247 - 1271)


This is the step we were missing - iterate over all "new" configurations 
and see if the property existed in the current stack's default configs as well.

If it existed in the current stack's _default_ configs but doesn't exist in 
the current configs, then it has no business being added back.


- Jonathan Hurley


On July 25, 2016, 5:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50409/
> ---
> 
> (Updated July 25, 2016, 5:04 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-17894
> https://issues.apache.org/jira/browse/AMBARI-17894
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> 
> - Install Ambari 2.2.2 with Hive on HDP 2.4.2.0 and Kerberize
> - Upgrade to Ambari 2.4.0
> - Upgrade to HDP 2.5
> - Distribute Keytabs
> - Add a new service
> 
> At this point, the UI flags Hive as having configurations which need 
> attention. The follow are all blank and are marked as required:
> 
> {code}
> hive-site/hive.server2.authentication.ldap.url
> hiveserver2-site/hive.conf.restricted.list
> hiveserver2-site/hive.security.authenticator.manager
> hiveserver2-site/hive.security.authorization.manager
> {code}
> 
> There are actually two problems here:
> 
> - The Kerberos wizard interprets the stack advisor "delete" attribute and 
> improperly sets config properties to blank instead of actually removing them
> 
> - The upgrade logic is adding properties back when it should not.
> 
> The Kerberos issue aside, we can't be adding properties back during upgrade 
> stack merging when those properties were specifically removed by the stack 
> advisor prior.
> 
> For example (Ambari 2.2.2 installed with HDP 2.x)
> - Ambari 2.2.2 does not have {{foo-site/property}} for HDP 2.x
> - Ambari 2.4.0 adds {{foo-site/property}} for HDP 2.x, but instructs the 
> upgrade not to add it
> - An upgrade to HDP 2.y sees that {{foo-site/property}} doesn't exist and 
> thinks it's brand new and needs to be merged.
> 
> The upgrade logic should check to see if {{foo-site/property}} is part of 
> both 2.x and 2.y default configurations. If it's part of both of them and is 
> not currently set, then upgrade should NOT set it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  55e9dbd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hiveserver2-site.xml
>  ad37114 
> 
> Diff: https://reviews.apache.org/r/50409/diff/
> 
> 
> Testing
> ---
> 
> PENDING UNIT TESTS
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>