Re: Review Request 64660: Rafactor: Package Log Search and Log Feeder into jars + cleanup start scripts

2017-12-17 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64660/#review194039
---


Ship it!




Ship It!

- Krisztian Kasa


On Dec. 16, 2017, 1:27 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64660/
> ---
> 
> (Updated Dec. 16, 2017, 1:27 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Krisztian Kasa, Miklos Gergely, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-22647
> https://issues.apache.org/jira/browse/AMBARI-22647
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - package logsearch / logfeeder classes into jars
> - create default logsearch-env and logfeeder-env files (those where only 
> generated by ambari stack code)
> - rename run.sh start scripts to logsearch.sh and logfeeder.sh, and create a 
> symlink for those in /usr/bin/
> (therefore we can call commands like: 'logsearch start' or 'logfeeder test 
> --test-log-entry ...')
> - refactor logfeeder command line tool -> new java entry point -> use it 
> through /usr/bin/logfeeder
> - remove pid / process handling logic from ambari stack code (as the new 
> logsaerch/logfeeder script will handle those)
> - move all config files from classes target/package/conf during maven package 
> phase
> - move "/etc/ambari-logsearch-.../conf" folder to 
> /usr/lib/ambari-logsearch.../conf, keep the old one as a symlink. (this 
> solution is useful as we can include every requried files in a tar.gz as well 
> and it can work without provided rpm/deb)
> - as conf file was moved out, we need to handle some cases during yum/apt 
> upgrade - move conf/keys/ or conf/checkpoints/ files to the new location (as 
> those could be generated there before and we do not want to loose them)
> - as conf files are moved, cleanup maven assembly configs
> - upgrade docker environment to make it work with the new changes
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-assembly/pom.xml cbc62ce 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/deb/logfeeder/postinst
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/deb/logfeeder/postrm
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/deb/logfeeder/posttrm
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/deb/logfeeder/preinst
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/deb/portal/postinst
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/deb/portal/postrm 
> 21a01fa 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/deb/portal/preinst
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/rpm/logfeeder/postinstall.sh
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/rpm/logfeeder/postremove.sh
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/rpm/logfeeder/preinstall.sh
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/rpm/portal/postinstall.sh
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/rpm/portal/postremove.sh
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-assembly/src/main/package/rpm/portal/preinstall.sh
>  PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/README.md d2e55f0 
>   ambari-logsearch/ambari-logsearch-logfeeder/build.properties 013ba2e 
>   ambari-logsearch/ambari-logsearch-logfeeder/build.xml 738b2ef 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 005af15 
>   ambari-logsearch/ambari-logsearch-logfeeder/run.sh 70947ec 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  2d31e5a 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java
>  61e7a1e 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  35c0e6a 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/package/deb/control/control
>  40cd855 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/package/deb/control/postinst
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/package/deb/control/postrm
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/package/deb/control/preinst
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/package/deb/control/prerm
>  21a01fa 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/scripts/logfeeder-env.sh 
> PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/src/main/scripts/logfeeder.sh 
>

Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory

2017-12-17 Thread Jonathan Hurley 


> On Dec. 17, 2017, 2:14 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> > Lines 155 (patched)
> > 
> >
> > OK - let's just make this a little less hard-coded ...
> > 
> > Can we pass in the paramters of livy/livy-server to this method and 
> > call it something like "fix_default_links"
> > 
> > We should also have some documentation as to why we need this method ...

Also, i think livy2 also had this


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64667/#review194033
---


On Dec. 16, 2017, 10:53 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64667/
> ---
> 
> (Updated Dec. 16, 2017, 10:53 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole.
> 
> 
> Bugs: AMBARI-22522
> https://issues.apache.org/jira/browse/AMBARI-22522
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Adding a small, livy-only workaround into install_packages.py
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> c8497cd 
> 
> 
> Diff: https://reviews.apache.org/r/64667/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test,
> live cluster check
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 64594: AMBARI-22649. Library for querying clusterSettings and stackSettings for its contents in command*.json

2017-12-17 Thread Attila Doroszlai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64594/#review194031
---


Fix it, then Ship it!




I've [attached a small unit 
test](https://issues.apache.org/jira/secure/attachment/12902568/AMBARI-22649-test.patch)
 for `settings.py` to 
[AMBARI-22649](https://issues.apache.org/jira/browse/AMBARI-22649), please feel 
free to use it.  I think it simplifies testing behavior of this library.


ambari-common/src/main/python/resource_management/libraries/functions/settings.py
Lines 28 (patched)


Default value of `None` for `setting_names` would make it easier to get all 
settings.



ambari-common/src/main/python/resource_management/libraries/functions/settings.py
Lines 47-49 (patched)


Shouldn't this also return `None`?  Now it says `/agentConfigParams` is not 
supported, but goes on to return the requested values.



ambari-common/src/main/python/resource_management/libraries/functions/settings.py
Lines 56 (patched)


Why is a `list`, `frozenset` or `tuple` not acceptable?



ambari-common/src/main/python/resource_management/libraries/functions/settings.py
Lines 60 (patched)


This is unreachable, since `None` is not an instance of `set`, hence it 
already returns `None` a bit earlier, instead of returning all settings here.



ambari-common/src/main/python/resource_management/libraries/functions/settings.py
Lines 69 (patched)


I think if the specified keys are not present, instead of `None` it should 
just return the empty `dict` that `result` already contains.

For example:

```
get_setting_type_entries('/clusterSettings', None)['key']
```

and 

```
get_setting_type_entries('/clusterSettings', set(['key']))['key']
```

should have the same result (actual value or `KeyError`), but returning 
`None` means the second one may result in `TypeError: 'NoneType' object has no 
attribute '__getitem__'` instead.



ambari-common/src/main/python/resource_management/libraries/functions/settings.py
Lines 89 (patched)


Should `return None` early if `setting_name is None`



ambari-common/src/main/python/resource_management/libraries/functions/settings.py
Lines 105-108 (patched)


Could be simplified to

```
return setting_type in (STACK_SETTINGS_TYPE, CLUSTER_SETTINGS_TYPE)
```


- Attila Doroszlai


On Dec. 17, 2017, 9:11 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64594/
> ---
> 
> (Updated Dec. 17, 2017, 9:11 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Dmytro Sen, Jayush Luniya, 
> Madhuvanthi Radhakrishnan, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22649
> https://issues.apache.org/jira/browse/AMBARI-22649
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced 
> "cluster settings" in Ambari.
> 
> **==**
> **Library for querying _clusterSettings_ and _stackSettings_ for its contents 
> in command*.json.**
> **==**
> 
> One should be able to query for a given **clusterSettings** or 
> **stackSettings**:
>  -  by passing in the setting name(one or more) in order to get it back as 
> key-value map, or
>  -  just get the value back for a passed-in setting.
> 
> 
> **Functions for clusterSettings:**
> **--**
>   - **get_cluster_setting_entries(setting_names)** : 
> -- Retrieves the passed-in cluster setting entr(y/ies) and their values 
> as a map.
>If 'setting_names' is passed-in as None : all the settings names and 
> their corresponding values will be returned as map.
>If 'setting_names' is passed-in as empty set : None will be returned.
> 
>   - **get_cluster_setting_value(setting_name)** :
> -- Retrieves the passed-in cluster setting entry's value.
> 
>   - **is_security_enabled()** : 
> -- Retrieves the cluster's security status.
> 
> 
> **Functions for stackSettings:** 
> ****
> 
> Stack settings as of now has 5 settings : stack_name, stack_root, 
> stack_features, stack_tools, stack_packages. stack_name, stack_root have 
> string as values,

Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory

2017-12-17 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64667/#review194033
---


Fix it, then Ship it!





ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
Lines 155 (patched)


OK - let's just make this a little less hard-coded ...

Can we pass in the paramters of livy/livy-server to this method and call it 
something like "fix_default_links"

We should also have some documentation as to why we need this method ...


- Jonathan Hurley


On Dec. 16, 2017, 10:53 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64667/
> ---
> 
> (Updated Dec. 16, 2017, 10:53 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole.
> 
> 
> Bugs: AMBARI-22522
> https://issues.apache.org/jira/browse/AMBARI-22522
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Adding a small, livy-only workaround into install_packages.py
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> c8497cd 
> 
> 
> Diff: https://reviews.apache.org/r/64667/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test,
> live cluster check
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory

2017-12-17 Thread Dmitro Lisnichenko 


> On Dec. 16, 2017, 10:35 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> > Lines 160-164 (patched)
> > 
> >
> > We should not hard code this into install_packages. Can we just expose 
> > this method for the Livy Python files to call?
> 
> Dmitro Lisnichenko wrote:
> In this case, this code should run before install_packages.py . 
> Otherwise, symlinks will be badly broken, and my code will not be able to fix 
> that. I can move the code to restart() of Livy, but we would have to require 
> users (via release notes) to restart Livy before installing packages for a 
> new stack. 
> Should I move this code to restart? Or should leave it as is, intending 
> to refactor/generalize it at 2.6.2?
> 
> Jonathan Hurley wrote:
> I don't see how they will be badly broken - we're updating the conf 
> pointers after the packages are installed anyway. Let's say that we have this 
> problem in the system:
> 
> /etc/livy/conf -> /usr/hdp/current/livy (broken)
> /usr/hdp/current/livy-server/conf -> /etc/component/conf (wrong)
> 
> During Livy's configure() or pre_upgrade_restart(), can we specifically 
> invoke conf_select.convert_conf_directories_to_symlinks() for the Livy 
> package?
> 
> ```
>   old_conf = directory_struct['conf_dir']
>   current_dir = directory_struct['current_dir']
>   if os.path.islink(old_conf):
> # it's already a link; make sure it's a link to where we want it
> if os.readlink(old_conf) != current_dir:
> ```
> 
> Seems like this would allow us to detect the broken link in 
> /etc/livy/conf and fix it.
> 
> Jonathan Hurley wrote:
> Ah, I dove deeper into this problem. To summarize:
> 
> /usr/hdp/current/livy-client/conf -> /etc/livy/conf exists on Ambari 2.5. 
> 
> After an upgrade to 2.6, we distribute packages for the new stack. The 
> code detects this problem (as it would on a first-time install) and tries to 
> fix it:
> 
> /etc/livy/conf -> /usr/hdp/current/livy-client/conf
> /usr/hdp/v1/livy/conf -> /etc/livy/conf
> /usr/hdp/v2/livy/conf -> /etc/livy/v2/0
> 
> Because we're only conf-selecting v2, v1 never gets correct since it's 
> already installed...

exactly


- Dmitro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64667/#review194019
---


On Dec. 16, 2017, 5:53 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64667/
> ---
> 
> (Updated Dec. 16, 2017, 5:53 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole.
> 
> 
> Bugs: AMBARI-22522
> https://issues.apache.org/jira/browse/AMBARI-22522
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Adding a small, livy-only workaround into install_packages.py
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> c8497cd 
> 
> 
> Diff: https://reviews.apache.org/r/64667/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test,
> live cluster check
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Review Request 64670: AMBARI-22660. Fix unit tests failing due to ServiceGroup issues

2017-12-17 Thread Attila Doroszlai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64670/
---

Review request for Ambari, Balázs Bence Sári, Dmytro Sen, Jayush Luniya, 
Madhuvanthi Radhakrishnan, Robert Nettleton, Swapan Shridhar, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-22660
https://issues.apache.org/jira/browse/AMBARI-22660


Repository: ambari


Description
---

Fixed mostly `ServiceGroupNotFound` due to `serviceGroupName=` and `Guice 
provision errors` due to `serviceGroup=null` in `ServiceFactory.createNew()`.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 25cfaf84c56b9b84f806ce43b9f8cd3fa4d36265 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 1bf5299e3c687403cbb8ba72b670929de5f53a12 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ServiceComponentHostRequest.java
 2554c2544875f70bb10432dc4bd5685f0b3c9643 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ServiceGroupDAO.java
 97659f74b5df2d837068644bce8dbd3e5319da1c 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostComponentStateEntity.java
 8994384956db0bac467cb61fd13afba07bbd21ab 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ServiceGroupEntity.java
 fedaee86fbbf4286b76a1c4758651f535bb6f642 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StackEntity.java
 c479cdd42742ebada75f57efafd764d6b2112148 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 bdda9a79ec952fda4ee64c46c6e2ed1c7c9e2d58 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java
 aff6d74f483ec365e3c8a96124f66e808c5b95ff 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java
 14ab8cf7baa26771f3b9eb72cf62518f9e93a7ce 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 0cc64e9d66b9afb21c62ab5b6346166d58e42036 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/BackgroundCustomCommandExecutionTest.java
 0b07fb7f1b31dde693116a97edc3a5425a82ba37 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/RefreshYarnCapacitySchedulerReleaseConfigTest.java
 99a7de370572ba055c31a056ea9012a66522 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AbstractResourceProviderTest.java
 0f04e744bad5f4005428dc018c9850aecdccc3d5 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ComponentResourceProviderTest.java
 be7134e7c51426e941b92b3ac8aaa9f9935c0095 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostComponentResourceProviderTest.java
 bad63b789463653df5eaaabf89a2b0b917cebdbf 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/JMXHostProviderTest.java
 d8575e7d6dfa9bda5671027bdf3ff152749a9ee9 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ServiceGroupResourceProviderTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ServiceResourceProviderTest.java
 3a239689f2f5b21528f9b47f0b3f26963dc2a2c6 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/logging/LoggingSearchPropertyProviderTest.java
 e9f7c4df07b623ac4c50e6869f80715b183fd45c 
  ambari-server/src/test/java/org/apache/ambari/server/events/EventsTest.java 
2376079af4f45d17eec0cc1e91bcf0ec6ab96b90 
  
ambari-server/src/test/java/org/apache/ambari/server/events/listeners/upgrade/HostVersionOutOfSyncListenerTest.java
 970b7cc6095f5607551fe66a5ea3792977582325 
  ambari-server/src/test/java/org/apache/ambari/server/orm/OrmTestHelper.java 
1a8340ab0aec52b6231177f34ecec32240ed7cbb 
  
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/AlertDispatchDAOTest.java
 b5525658c2b45a2ef3ba649e66ff95b673990519 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ComponentVersionCheckActionTest.java
 3ceb6fa1d6bbb217b70cbc64e8687d74f08cf85e 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ConfigureActionTest.java
 cc2bba7f4c6bdf930c0edac19810caf645759675 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureActionTest.java
 1bad2199d07017721d021c646cf8c87d604f48ea 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeActionTest.java
 e9da5641a8ad0e44f9b04c2cf035635bafa7d535 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ServiceComponentTest.java
 f4c3d20b435e048a60d64a874db210fa9a033bc6 
  
ambari-server/src/test/java/org/apache/ambari/server/state/alerts/AlertEventPublisherTest.java
 50555d09ac58572627cec7d98ffa2a452e74cbc0

Re: Review Request 64594: AMBARI-22649. Library for querying clusterSettings and stackSettings for its contents in command*.json

2017-12-17 Thread Swapan Shridhar 


> On Dec. 16, 2017, 12:49 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_settings.py
> > Lines 32-88 (patched)
> > 
> >
> > Most of these 2 functions are the same as those in `cluster_settings`.  
> > Can you please try to reduce duplication?

Done.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64594/#review194013
---


On Dec. 17, 2017, 8:11 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64594/
> ---
> 
> (Updated Dec. 17, 2017, 8:11 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Dmytro Sen, Jayush Luniya, 
> Madhuvanthi Radhakrishnan, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22649
> https://issues.apache.org/jira/browse/AMBARI-22649
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced 
> "cluster settings" in Ambari.
> 
> **==**
> **Library for querying _clusterSettings_ and _stackSettings_ for its contents 
> in command*.json.**
> **==**
> 
> One should be able to query for a given **clusterSettings** or 
> **stackSettings**:
>  -  by passing in the setting name(one or more) in order to get it back as 
> key-value map, or
>  -  just get the value back for a passed-in setting.
> 
> 
> **Functions for clusterSettings:**
> **--**
>   - **get_cluster_setting_entries(setting_names)** : 
> -- Retrieves the passed-in cluster setting entr(y/ies) and their values 
> as a map.
>If 'setting_names' is passed-in as None : all the settings names and 
> their corresponding values will be returned as map.
>If 'setting_names' is passed-in as empty set : None will be returned.
> 
>   - **get_cluster_setting_value(setting_name)** :
> -- Retrieves the passed-in cluster setting entry's value.
> 
>   - **is_security_enabled()** : 
> -- Retrieves the cluster's security status.
> 
> 
> **Functions for stackSettings:** 
> ****
> 
> Stack settings as of now has 5 settings : stack_name, stack_root, 
> stack_features, stack_tools, stack_packages. stack_name, stack_root have 
> string as values, whereas stack_features, stack_tools, stack_packages have 
> values as JSON. Further there already exists python functions in files : 
> **stack_features.py**, **stack_tools.py** and **stack_select.py**.
> 
>- **get_stack_setting_entries(setting_names)** : 
>   --   Retrieves the passed-in stack setting entr(y/ies) and their values 
> as a map.
> If 'setting_names' is passed-in as None, all the settings names 
> and their corresponding values will be returned as map.
> If 'setting_names' is passed-in as empty set : None will be 
> returned.
> 
>- **get_stack_setting_value(setting_name)**:
> -- Retrieves the passed-in stack setting entry's value.
> 
> - **get_stack_name()**:
> -- Retrieves the stack name.
> 
> - **get_stack_root()**:
>-- Retrieves the stack root.
> 
>  
> 
> **Modifications in  _stack_features.py, stack_tools.py and stack_select.py_ 
> files:**
> **-**
> 
> - Given that these already exist and as of now they read the relevant stack 
> setting from *configurations/cluster_env*. 
> - Thus, code has been added to try reading from /stackSettings first by 
> calling the new fn.() get_stack_setting_value(). if setting not found, go for 
> the fall back  *configurations/cluster_env* (which would be removed soon, 
> when we remove cluster_env).
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/cluster_settings.py
>  PRE-CREATION 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/settings.py
>  PRE-CREATION 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  92823b0 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  b741a33 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_settings.py
>  PRE-CREATION 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
>  d9233a3 
> 
> 
> Diff: https://reviews.apache.org/r/64594/diff/3/
> 
> 
> Testing
> ---
> 
> Python UT passes.
> 
> 
> **Testing:**
> 
> Tested on live

Re: Review Request 64594: AMBARI-22649. Library for querying clusterSettings and stackSettings for its contents in command*.json

2017-12-17 Thread Swapan Shridhar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64594/
---

(Updated Dec. 17, 2017, 8:11 a.m.)


Review request for Ambari, Attila Doroszlai, Dmytro Sen, Jayush Luniya, 
Madhuvanthi Radhakrishnan, and Vitalyi Brodetskyi.


Changes
---

Updated cluster_settings.py and stack_settings.py code to common location 
settings.py


Bugs: AMBARI-22649
https://issues.apache.org/jira/browse/AMBARI-22649


Repository: ambari


Description
---

Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced 
"cluster settings" in Ambari.

**==**
**Library for querying _clusterSettings_ and _stackSettings_ for its contents 
in command*.json.**
**==**

One should be able to query for a given **clusterSettings** or 
**stackSettings**:
 -  by passing in the setting name(one or more) in order to get it back as 
key-value map, or
 -  just get the value back for a passed-in setting.


**Functions for clusterSettings:**
**--**
  - **get_cluster_setting_entries(setting_names)** : 
-- Retrieves the passed-in cluster setting entr(y/ies) and their values as 
a map.
   If 'setting_names' is passed-in as None : all the settings names and 
their corresponding values will be returned as map.
   If 'setting_names' is passed-in as empty set : None will be returned.

  - **get_cluster_setting_value(setting_name)** :
-- Retrieves the passed-in cluster setting entry's value.

  - **is_security_enabled()** : 
-- Retrieves the cluster's security status.


**Functions for stackSettings:** 
****

Stack settings as of now has 5 settings : stack_name, stack_root, 
stack_features, stack_tools, stack_packages. stack_name, stack_root have string 
as values, whereas stack_features, stack_tools, stack_packages have values as 
JSON. Further there already exists python functions in files : 
**stack_features.py**, **stack_tools.py** and **stack_select.py**.

   - **get_stack_setting_entries(setting_names)** : 
  --   Retrieves the passed-in stack setting entr(y/ies) and their values 
as a map.
If 'setting_names' is passed-in as None, all the settings names and 
their corresponding values will be returned as map.
If 'setting_names' is passed-in as empty set : None will be 
returned.

   - **get_stack_setting_value(setting_name)**:
-- Retrieves the passed-in stack setting entry's value.

- **get_stack_name()**:
-- Retrieves the stack name.

- **get_stack_root()**:
   -- Retrieves the stack root.

 

**Modifications in  _stack_features.py, stack_tools.py and stack_select.py_ 
files:**
**-**

- Given that these already exist and as of now they read the relevant stack 
setting from *configurations/cluster_env*. 
- Thus, code has been added to try reading from /stackSettings first by calling 
the new fn.() get_stack_setting_value(). if setting not found, go for the fall 
back  *configurations/cluster_env* (which would be removed soon, when we remove 
cluster_env).


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/cluster_settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 92823b0 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 b741a33 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
 d9233a3 


Diff: https://reviews.apache.org/r/64594/diff/3/

Changes: https://reviews.apache.org/r/64594/diff/2-3/


Testing
---

Python UT passes.


**Testing:**

Tested on live cluster


**=**
**clusterSettings:**
**=**

**A.** get_cluster_setting_entries():
**--**

  - 1. Retrieve **single** setting : 'recovery_enabled'
-- In get_cluster_setting_entries(). Passed-in setting(s) : 
set(['recovery_enabled'])
  **o/p**:   {'recovery_enabled': True}

   - 2. Retrieve **two** settings : 'recovery_enabled', 'sysprep_skip_setup_jce'
-- In get_cluster_setting_entries(). Passed-in setting(s) : 
set(['recovery_enabled', 'sysprep_skip_setup_jce'])
**o/p**:   {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}


- 3. Retrieve settings where passed in empty set -> Expected nothing returned
 -- In get_cluster_setting_entries(). Passed-in setting(s) : set([])