Review Request 64709: Stack unit tests fixes

2017-12-19 Thread Andrew Onischuk 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64709/
---

Review request for Ambari, Attila Doroszlai, Myroslav Papirkovskyy, Sid Wagle, 
and Vitalyi Brodetskyi.


Bugs: AMBARI-22671
https://issues.apache.org/jira/browse/AMBARI-22671


Repository: ambari


Description
---

Fix ~100 UT.

Before:



Total run:1210
Total errors:195
Total failures:44


After:



Total run:1210
Total errors:83
Total failures:67


Diffs
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
21aeec7200 
  ambari-server/src/test/python/common-services/configs/hive_default.json 
45af84788d 
  
ambari-server/src/test/python/common-services/configs/hive_unsupported_jdbc_type.json
 76759c1d58 
  ambari-server/src/test/python/common-services/configs/pxf_default.json 
384ca40614 
  
ambari-server/src/test/python/common-services/configs/ranger_admin_default.json 
4702d65177 
  
ambari-server/src/test/python/common-services/configs/ranger_admin_unsupported_db_flavor.json
 d7abe6b156 
  ambari-server/src/test/python/common-services/configs/sqoop_default.json 
0c74006f37 
  
ambari-server/src/test/python/common-services/configs/sqoop_unsupported_jdbc_driver.json
 b571b9a7a3 
  
ambari-server/src/test/python/custom_actions/configs/install_packages_config.json
 62ce5f1910 
  
ambari-server/src/test/python/custom_actions/configs/install_packages_repository_file.json
 84bc64d743 
  
ambari-server/src/test/python/custom_actions/configs/remove_previous_stacks.json
 ec28e05558 
  ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 
e153225f55 
  ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json 
f1d1944fcf 
  ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json 
a112b64652 
  ambari-server/src/test/python/stacks/2.0.6/configs/default.hbasedecom.json 
2ec7f519b0 
  ambari-server/src/test/python/stacks/2.0.6/configs/default.json d5c676e8b5 
  
ambari-server/src/test/python/stacks/2.0.6/configs/default.non_gmetad_host.json 
b65d3c0743 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_ams_embedded.json 
dabedf7cc3 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_client.json 
b6a97b9311 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json 
9bac43f386 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 
da46ad3b04 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 
04fc65259d 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 
cb026da863 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 
919a7cf564 
  
ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json
 8e33fce1ab 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_with_bucket.json 
7a6da349df 
  
ambari-server/src/test/python/stacks/2.0.6/configs/default_yarn_include_file_dont_manage.json
 10c0827d3f 
  
ambari-server/src/test/python/stacks/2.0.6/configs/default_yarn_include_file_manage.json
 d896d8ea1b 
  ambari-server/src/test/python/stacks/2.0.6/configs/flume_target.json 
06ed171cd9 
  
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json
 c90b6b2558 
  
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json
 65b2907cb1 
  
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node_initial_start.json
 d9ecd86ae4 
  
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node_initial_start_dfs_nameservices.json
 c02aa2f946 
  ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json c50ddb6f74 
  ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 243e2500f5 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json b4f203765f 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 
2892446562 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-preupgrade.json 
d30aac8b92 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2-phoenix.json 
27874a8e26 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 
7a35fbdbe0 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase_no_phx.json 
27c813bb4a 
  ambari-server/src/test/python/stacks/2.0.6/configs/hbase_with_phx.json 
2350bdf82e 
  ambari-server/src/test/python/stacks/2.0.6/configs/nn_ru_lzo.json ecac966345 
  ambari-server/src/test/python/stacks/2.0.6/configs/oozie_existing_sqla.json 
40d0ca8b61 
  ambari-server/src/test/python/stacks/2.0.6/configs/rebalancehdfs_default.json 
02a34f352a 
  ambari-server/src/test/python/stacks/2.0.6/configs/rebalancehdfs_secured.json 
dc097a103c 
  ambari-server/src/test/python/stacks/2.0.6/configs/repository_file.json 
e9beea9d33

Re: Review Request 64709: Stack unit tests fixes

2017-12-19 Thread Vitalyi Brodetskyi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64709/#review194140
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Гру. 19, 2017, 12:02 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64709/
> ---
> 
> (Updated Гру. 19, 2017, 12:02 після полудня)
> 
> 
> Review request for Ambari, Attila Doroszlai, Myroslav Papirkovskyy, Sid 
> Wagle, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22671
> https://issues.apache.org/jira/browse/AMBARI-22671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix ~100 UT.
> 
> Before:
> 
> 
> 
> Total run:1210
> Total errors:195
> Total failures:44
> 
> 
> After:
> 
> 
> 
> Total run:1210
> Total errors:83
> Total failures:67
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 21aeec7200 
>   ambari-server/src/test/python/common-services/configs/hive_default.json 
> 45af84788d 
>   
> ambari-server/src/test/python/common-services/configs/hive_unsupported_jdbc_type.json
>  76759c1d58 
>   ambari-server/src/test/python/common-services/configs/pxf_default.json 
> 384ca40614 
>   
> ambari-server/src/test/python/common-services/configs/ranger_admin_default.json
>  4702d65177 
>   
> ambari-server/src/test/python/common-services/configs/ranger_admin_unsupported_db_flavor.json
>  d7abe6b156 
>   ambari-server/src/test/python/common-services/configs/sqoop_default.json 
> 0c74006f37 
>   
> ambari-server/src/test/python/common-services/configs/sqoop_unsupported_jdbc_driver.json
>  b571b9a7a3 
>   
> ambari-server/src/test/python/custom_actions/configs/install_packages_config.json
>  62ce5f1910 
>   
> ambari-server/src/test/python/custom_actions/configs/install_packages_repository_file.json
>  84bc64d743 
>   
> ambari-server/src/test/python/custom_actions/configs/remove_previous_stacks.json
>  ec28e05558 
>   ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 
> e153225f55 
>   ambari-server/src/test/python/stacks/2.0.6/configs/altfs_plus_hdfs.json 
> f1d1944fcf 
>   ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json 
> a112b64652 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default.hbasedecom.json 
> 2ec7f519b0 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default.json d5c676e8b5 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default.non_gmetad_host.json
>  b65d3c0743 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_ams_embedded.json 
> dabedf7cc3 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default_client.json 
> b6a97b9311 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha.json 
> 9bac43f386 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_nn_ha_2.json 
> da46ad3b04 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 
> 04fc65259d 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json 
> cb026da863 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default_oozie_mysql.json 
> 919a7cf564 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_update_exclude_file_only.json
>  8e33fce1ab 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default_with_bucket.json 
> 7a6da349df 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_yarn_include_file_dont_manage.json
>  10c0827d3f 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_yarn_include_file_manage.json
>  d896d8ea1b 
>   ambari-server/src/test/python/stacks/2.0.6/configs/flume_target.json 
> 06ed171cd9 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json
>  c90b6b2558 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json
>  65b2907cb1 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node_initial_start.json
>  d9ecd86ae4 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node_initial_start_dfs_nameservices.json
>  c02aa2f946 
>   ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 
> c50ddb6f74 
>   ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 
> 243e2500f5 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json 
> b4f203765f 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 
> 2892446562 
>   ambari-server/src/test/python/stacks/2.0.6/configs/hbase-preupgrade.json 
> d30aac8b92 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2-phoenix.json 
> 27874a8e26 
>

Review Request 64710: UI hangs on step 2 of security wizard

2017-12-19 Thread Andrew Onischuk 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64710/
---

Review request for Ambari, Myroslav Papirkovskyy, Sid Wagle, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-22673
https://issues.apache.org/jira/browse/AMBARI-22673


Repository: ambari


Description
---


Diffs
-

  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/properties/krb5_conf.j2
 PRE-CREATION 


Diff: https://reviews.apache.org/r/64710/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 64594: AMBARI-22649. Library for querying clusterSettings and stackSettings for its contents in command*.json

2017-12-19 Thread Swapan Shridhar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64594/
---

(Updated Dec. 19, 2017, 2:09 p.m.)


Review request for Ambari, Attila Doroszlai, Dmytro Sen, Jayush Luniya, 
Madhuvanthi Radhakrishnan, and Vitalyi Brodetskyi.


Changes
---

Updated review as per Attila's comments. Furhher, Attila had provided UT code, 
which I improvized further and added to this review request. Thanks Attila for 
that.


Bugs: AMBARI-22649
https://issues.apache.org/jira/browse/AMBARI-22649


Repository: ambari


Description
---

Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced 
"cluster settings" in Ambari.

**==**
**Library for querying _clusterSettings_ and _stackSettings_ for its contents 
in command*.json.**
**==**

One should be able to query for a given **clusterSettings** or 
**stackSettings**:
 -  by passing in the setting name(one or more) in order to get it back as 
key-value map, or
 -  just get the value back for a passed-in setting.


**Functions for clusterSettings:**
**--**
  - **get_cluster_setting_entries(setting_names)** : 
-- Retrieves the passed-in cluster setting entr(y/ies) and their values as 
a map.
   If 'setting_names' is passed-in as None : all the settings names and 
their corresponding values will be returned as map.
   If 'setting_names' is passed-in as empty set : None will be returned.

  - **get_cluster_setting_value(setting_name)** :
-- Retrieves the passed-in cluster setting entry's value.

  - **is_security_enabled()** : 
-- Retrieves the cluster's security status.


**Functions for stackSettings:** 
****

Stack settings as of now has 5 settings : stack_name, stack_root, 
stack_features, stack_tools, stack_packages. stack_name, stack_root have string 
as values, whereas stack_features, stack_tools, stack_packages have values as 
JSON. Further there already exists python functions in files : 
**stack_features.py**, **stack_tools.py** and **stack_select.py**.

   - **get_stack_setting_entries(setting_names)** : 
  --   Retrieves the passed-in stack setting entr(y/ies) and their values 
as a map.
If 'setting_names' is passed-in as None, all the settings names and 
their corresponding values will be returned as map.
If 'setting_names' is passed-in as empty set : None will be 
returned.

   - **get_stack_setting_value(setting_name)**:
-- Retrieves the passed-in stack setting entry's value.

- **get_stack_name()**:
-- Retrieves the stack name.

- **get_stack_root()**:
   -- Retrieves the stack root.

 

**Modifications in  _stack_features.py, stack_tools.py and stack_select.py_ 
files:**
**-**

- Given that these already exist and as of now they read the relevant stack 
setting from *configurations/cluster_env*. 
- Thus, code has been added to try reading from /stackSettings first by calling 
the new fn.() get_stack_setting_value(). if setting not found, go for the fall 
back  *configurations/cluster_env* (which would be removed soon, when we remove 
cluster_env).


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/cluster_settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 92823b0 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 b741a33 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
 d9233a3 
  ambari-server/src/test/python/TestSettings.py PRE-CREATION 


Diff: https://reviews.apache.org/r/64594/diff/4/

Changes: https://reviews.apache.org/r/64594/diff/3-4/


Testing
---

Python UT passes.


**Testing:**

Tested on live cluster


**=**
**clusterSettings:**
**=**

**A.** get_cluster_setting_entries():
**--**

  - 1. Retrieve **single** setting : 'recovery_enabled'
-- In get_cluster_setting_entries(). Passed-in setting(s) : 
set(['recovery_enabled'])
  **o/p**:   {'recovery_enabled': True}

   - 2. Retrieve **two** settings : 'recovery_enabled', 'sysprep_skip_setup_jce'
-- In get_cluster_setting_entries(). Passed-in setting(s) : 
set(['recovery_enabled', 'sysprep_skip_setup_jce'])
**o/p**:   {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}


- 3.

Re: Review Request 64594: AMBARI-22649. Library for querying clusterSettings and stackSettings for its contents in command*.json

2017-12-19 Thread Swapan Shridhar 


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > I've [attached a small unit 
> > test](https://issues.apache.org/jira/secure/attachment/12902568/AMBARI-22649-test.patch)
> >  for `settings.py` to 
> > [AMBARI-22649](https://issues.apache.org/jira/browse/AMBARI-22649), please 
> > feel free to use it.  I think it simplifies testing behavior of this 
> > library.

Thanks Attila. This helps.


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/settings.py
> > Lines 28 (patched)
> > 
> >
> > Default value of `None` for `setting_names` would make it easier to get 
> > all settings.

Done and UT added.


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/settings.py
> > Lines 47-49 (patched)
> > 
> >
> > Shouldn't this also return `None`?  Now it says `/agentConfigParams` is 
> > not supported, but goes on to return the requested values.

Fixed.


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/settings.py
> > Lines 56 (patched)
> > 
> >
> > Why is a `list`, `frozenset` or `tuple` not acceptable?

Added support for them.


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/settings.py
> > Lines 60 (patched)
> > 
> >
> > This is unreachable, since `None` is not an instance of `set`, hence it 
> > already returns `None` a bit earlier, instead of returning all settings 
> > here.

Fixed.


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/settings.py
> > Lines 69 (patched)
> > 
> >
> > I think if the specified keys are not present, instead of `None` it 
> > should just return the empty `dict` that `result` already contains.
> > 
> > For example:
> > 
> > ```
> > get_setting_type_entries('/clusterSettings', None)['key']
> > ```
> > 
> > and 
> > 
> > ```
> > get_setting_type_entries('/clusterSettings', set(['key']))['key']
> > ```
> > 
> > should have the same result (actual value or `KeyError`), but returning 
> > `None` means the second one may result in `TypeError: 'NoneType' object has 
> > no attribute '__getitem__'` instead.

Fixed.


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/settings.py
> > Lines 89 (patched)
> > 
> >
> > Should `return None` early if `setting_name is None`

Fixed.


> On Dec. 17, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/settings.py
> > Lines 105-108 (patched)
> > 
> >
> > Could be simplified to
> > 
> > ```
> > return setting_type in (STACK_SETTINGS_TYPE, CLUSTER_SETTINGS_TYPE)
> > ```

Fixed.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64594/#review194031
---


On Dec. 17, 2017, 8:11 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64594/
> ---
> 
> (Updated Dec. 17, 2017, 8:11 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Dmytro Sen, Jayush Luniya, 
> Madhuvanthi Radhakrishnan, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22649
> https://issues.apache.org/jira/browse/AMBARI-22649
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced 
> "cluster settings" in Ambari.
> 
> **==**
> **Library for querying _clusterSettings_ and _stackSettings_ for its contents 
> in command*.json.**
> **==**
> 
> One should be able to query for a given **clusterSettings** or 
> **stackSettings**:
>  -  by passing in the setting name(one or more) in order to get it back as 
> key-value map, or
>  -  just

Re: Review Request 64594: AMBARI-22649. Library for querying clusterSettings and stackSettings for its contents in command*.json

2017-12-19 Thread Swapan Shridhar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64594/
---

(Updated Dec. 19, 2017, 2:10 p.m.)


Review request for Ambari, Attila Doroszlai, Dmytro Sen, Jayush Luniya, 
Madhuvanthi Radhakrishnan, and Vitalyi Brodetskyi.


Changes
---

Added "Refer **TestSettings.py** for usage."


Bugs: AMBARI-22649
https://issues.apache.org/jira/browse/AMBARI-22649


Repository: ambari


Description
---

Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced 
"cluster settings" in Ambari.

**==**
**Library for querying _clusterSettings_ and _stackSettings_ for its contents 
in command*.json.**
**==**

One should be able to query for a given **clusterSettings** or 
**stackSettings**:
 -  by passing in the setting name(one or more) in order to get it back as 
key-value map, or
 -  just get the value back for a passed-in setting.


**Functions for clusterSettings:**
**--**
  - **get_cluster_setting_entries(setting_names)** : 
-- Retrieves the passed-in cluster setting entr(y/ies) and their values as 
a map.
   If 'setting_names' is passed-in as None : all the settings names and 
their corresponding values will be returned as map.
   If 'setting_names' is passed-in as empty set : None will be returned.

  - **get_cluster_setting_value(setting_name)** :
-- Retrieves the passed-in cluster setting entry's value.

  - **is_security_enabled()** : 
-- Retrieves the cluster's security status.


**Functions for stackSettings:** 
****

Stack settings as of now has 5 settings : stack_name, stack_root, 
stack_features, stack_tools, stack_packages. stack_name, stack_root have string 
as values, whereas stack_features, stack_tools, stack_packages have values as 
JSON. Further there already exists python functions in files : 
**stack_features.py**, **stack_tools.py** and **stack_select.py**.

   - **get_stack_setting_entries(setting_names)** : 
  --   Retrieves the passed-in stack setting entr(y/ies) and their values 
as a map.
If 'setting_names' is passed-in as None, all the settings names and 
their corresponding values will be returned as map.
If 'setting_names' is passed-in as empty set : None will be 
returned.

   - **get_stack_setting_value(setting_name)**:
-- Retrieves the passed-in stack setting entry's value.

- **get_stack_name()**:
-- Retrieves the stack name.

- **get_stack_root()**:
   -- Retrieves the stack root.

 

**Modifications in  _stack_features.py, stack_tools.py and stack_select.py_ 
files:**
**-**

- Given that these already exist and as of now they read the relevant stack 
setting from *configurations/cluster_env*. 
- Thus, code has been added to try reading from /stackSettings first by calling 
the new fn.() get_stack_setting_value(). if setting not found, go for the fall 
back  *configurations/cluster_env* (which would be removed soon, when we remove 
cluster_env).


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/cluster_settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 92823b0 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 b741a33 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_settings.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
 d9233a3 
  ambari-server/src/test/python/TestSettings.py PRE-CREATION 


Diff: https://reviews.apache.org/r/64594/diff/4/


Testing (updated)
---

Python UT passes.


  Refer **TestSettings.py** for usage.


**Testing:**

Tested on live cluster


**=**
**clusterSettings:**
**=**

**A.** get_cluster_setting_entries():
**--**

  - 1. Retrieve **single** setting : 'recovery_enabled'
-- In get_cluster_setting_entries(). Passed-in setting(s) : 
set(['recovery_enabled'])
  **o/p**:   {'recovery_enabled': True}

   - 2. Retrieve **two** settings : 'recovery_enabled', 'sysprep_skip_setup_jce'
-- In get_cluster_setting_entries(). Passed-in setting(s) : 
set(['recovery_enabled', 'sysprep_skip_setup_jce'])
**o/p**:   {'recovery_enabled': True, 'sysprep_skip_setup_jce': False}


- 3. Retrieve settings where passed in empty set -> Expected nothing returned
 -- In get_cluster_setting_entries(). Passed-in setting(s) :

Re: Review Request 64693: Add service group dependencies

2017-12-19 Thread Jayush Luniya 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64693/#review194150
---



Unit tests?

- Jayush Luniya


On Dec. 18, 2017, 11:53 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64693/
> ---
> 
> (Updated Dec. 18, 2017, 11:53 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Madhuvanthi Radhakrishnan, and 
> Swapan Shridhar.
> 
> 
> Bugs: AMBARI-22249
> https://issues.apache.org/jira/browse/AMBARI-22249
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Chnaged adding service group dependency by name, not by id. Fixed 
> add/get/delete requests.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ServiceGroupKey.java
>  9389f0c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ServiceGroupDependencyRequest.java
>  9f4eea3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ServiceGroupDependencyResponse.java
>  1a4b4ed 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceGroupDependencyResourceProvider.java
>  aac7102 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ServiceGroupDependencyEntity.java
>  ee0beeb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceGroupImpl.java
>  914b65a 
> 
> 
> Diff: https://reviews.apache.org/r/64693/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Review Request 64716: LogSearch Title Bar Fixes

2017-12-19 Thread Andrii Babiichuk 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64716/
---

Review request for Ambari and Aleksandr Kovalenko.


Bugs: AMBARI-22675
https://issues.apache.org/jira/browse/AMBARI-22675


Repository: ambari


Description
---

The top LogSearch Title bar is currently fixed in size and shown during 
scrolling. It's better to have a limited set of search controls displayed in 
the LogSearch title bar, to provide a way to show the existing search, and also 
allow the user to modify the search while in the middle of searching results.
The search bar instead of the title bar should be sticky at the top.


Diffs
-

  ambari-logsearch/ambari-logsearch-web/src/app/components/app.component.html 
833f43f 
  ambari-logsearch/ambari-logsearch-web/src/app/components/app.component.less 
f0fecfc 
  
ambari-logsearch/ambari-logsearch-web/src/app/components/filters-panel/filters-panel.component.ts
 f9fe94b 
  
ambari-logsearch/ambari-logsearch-web/src/app/components/logs-container/logs-container.component.html
 5e401d8 
  
ambari-logsearch/ambari-logsearch-web/src/app/components/logs-container/logs-container.component.less
 b4d44fb 
  
ambari-logsearch/ambari-logsearch-web/src/app/components/logs-container/logs-container.component.ts
 86709fb 
  
ambari-logsearch/ambari-logsearch-web/src/app/components/main-container/main-container.component.html
 95dd238 
  
ambari-logsearch/ambari-logsearch-web/src/app/components/main-container/main-container.component.less
 bca668d 
  
ambari-logsearch/ambari-logsearch-web/src/app/components/main-container/main-container.component.ts
 6747a0c 


Diff: https://reviews.apache.org/r/64716/diff/1/


Testing
---

Executed 221 of 221 SUCCESS (7.013 secs / 6.895 secs)


Thanks,

Andrii Babiichuk

Re: Review Request 64716: LogSearch Title Bar Fixes

2017-12-19 Thread Aleksandr Kovalenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64716/#review194151
---


Ship it!




Ship It!

- Aleksandr Kovalenko


On Дек. 19, 2017, 5:15 п.п., Andrii Babiichuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64716/
> ---
> 
> (Updated Дек. 19, 2017, 5:15 п.п.)
> 
> 
> Review request for Ambari and Aleksandr Kovalenko.
> 
> 
> Bugs: AMBARI-22675
> https://issues.apache.org/jira/browse/AMBARI-22675
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The top LogSearch Title bar is currently fixed in size and shown during 
> scrolling. It's better to have a limited set of search controls displayed in 
> the LogSearch title bar, to provide a way to show the existing search, and 
> also allow the user to modify the search while in the middle of searching 
> results.
> The search bar instead of the title bar should be sticky at the top.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-web/src/app/components/app.component.html 
> 833f43f 
>   ambari-logsearch/ambari-logsearch-web/src/app/components/app.component.less 
> f0fecfc 
>   
> ambari-logsearch/ambari-logsearch-web/src/app/components/filters-panel/filters-panel.component.ts
>  f9fe94b 
>   
> ambari-logsearch/ambari-logsearch-web/src/app/components/logs-container/logs-container.component.html
>  5e401d8 
>   
> ambari-logsearch/ambari-logsearch-web/src/app/components/logs-container/logs-container.component.less
>  b4d44fb 
>   
> ambari-logsearch/ambari-logsearch-web/src/app/components/logs-container/logs-container.component.ts
>  86709fb 
>   
> ambari-logsearch/ambari-logsearch-web/src/app/components/main-container/main-container.component.html
>  95dd238 
>   
> ambari-logsearch/ambari-logsearch-web/src/app/components/main-container/main-container.component.less
>  bca668d 
>   
> ambari-logsearch/ambari-logsearch-web/src/app/components/main-container/main-container.component.ts
>  6747a0c 
> 
> 
> Diff: https://reviews.apache.org/r/64716/diff/1/
> 
> 
> Testing
> ---
> 
> Executed 221 of 221 SUCCESS (7.013 secs / 6.895 secs)
> 
> 
> Thanks,
> 
> Andrii Babiichuk
> 
>

Re: Review Request 64118: Refactor internal code of handling info between kerberos wizard actions

2017-12-19 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64118/#review194176
---


Ship it!




Ship It!

- Robert Levas


On Dec. 13, 2017, 1:14 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64118/
> ---
> 
> (Updated Dec. 13, 2017, 1:14 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, and Robert Levas.
> 
> 
> Bugs: AMBARI-22530
> https://issues.apache.org/jira/browse/AMBARI-22530
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Moved out of data files for principal generation.
> Now before every kerberos action all info written to database.
> 
> Server actions determine what principals to process based on filters passed 
> to action, this allow:
> 
> * selective generation
> * split generation process per-host/per-service/per-component basis and make 
> generation concurrent for seperate parts
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  53cceb0ded 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  83d2c9808d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  a7b9d80df0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  749943dc00 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  ab85aa1d7c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostKerberosIdentityResourceProvider.java
>  52ab9b56ae 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java
>  a8723b7bfa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabPrincipalDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalDAO.java
>  81e4b3d0bf 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalHostDAO.java
>  f27dc48ea4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostGroupComponentEntityPK.java
>  0898133bb5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabEntity.java
>  a25931b946 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabPrincipalEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabServiceMappingEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalEntity.java
>  5dd54ca30c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntity.java
>  d4e80c65d2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntityPK.java
>  7e57e4ae95 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  b8affb4e19 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CleanupServerAction.java
>  002076d85c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/ConfigureAmbariIdentitiesServerAction.java
>  338415280f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java
>  5ec4c1011e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java
>  0c906592af 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/DestroyPrincipalsServerAction.java
>  4c80bd425e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/FinalizeKerberosServerAction.java
>  bfd5e4036d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java
>  ff5f5cef1c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareEnableKerberosServerAction.java
>  671ad95c8f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareKerberosIdentitiesServerAction.java
>  83a2106afd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/KerberosKeytabController.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosKeytab.java
>  17e484ad10 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosPrincipal.java
>  PRE-CREATION 
>   
>

Review Request 64722: [Patch Hive]webhcat: test_sqoop fails with hdfs:///hdp/apps/2.6.*/sqoop/sqoop.tar.gz does not exist

2017-12-19 Thread Nate Cole 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64722/
---

Review request for Ambari and Jonathan Hurley.


Bugs: AMBARI-22676
https://issues.apache.org/jira/browse/AMBARI-22676


Repository: ambari


Description
---

Add SQOOP to the list of services that should go with HIVE


Diffs
-

  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
 62a46b91bd 


Diff: https://reviews.apache.org/r/64722/diff/1/


Testing
---

No new tests for json change.


Thanks,

Nate Cole

Re: Review Request 64722: [Patch Hive]webhcat: test_sqoop fails with hdfs:///hdp/apps/2.6.*/sqoop/sqoop.tar.gz does not exist

2017-12-19 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64722/#review194185
---


Ship it!




Ship It!

- Jonathan Hurley


On Dec. 19, 2017, 3:58 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64722/
> ---
> 
> (Updated Dec. 19, 2017, 3:58 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan 
> Hurley.
> 
> 
> Bugs: AMBARI-22676
> https://issues.apache.org/jira/browse/AMBARI-22676
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add SQOOP to the list of services that should go with HIVE
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  62a46b91bd 
> 
> 
> Diff: https://reviews.apache.org/r/64722/diff/1/
> 
> 
> Testing
> ---
> 
> No new tests for json change.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 64544: Migrate user data for upgrade to improved user account management

2017-12-19 Thread Robert Levas 


> On Dec. 12, 2017, 11:44 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
> > Lines 175-192 (patched)
> > 
> >
> > Interesting .. is this idempotent? Typically nested-selects with 
> > inserts are not if they could violate PKs. Maybe you need to breakt this 
> > out into 2 queries and ensure you're only inserting when necessary?
> > 
> > Then again, maybe I don't quite see what this query is doing and it's 
> > ok... Thought I'd flag it for discussion.
> 
> Robert Levas wrote:
> I assumed that this would fail if run multiple times. This is why there 
> is a conditional right before it:
> ```
> // Migrate data from users table (if not yet upgraded)
> if (!usersTableUpgraded()) {
>   dbAccessor.executeUpdate(
>   "insert into " + USER_AUTHENTICATION_TABLE +
>   "(" + USER_AUTHENTICATION_USER_AUTHENTICATION_ID_COLUMN + 
> ", " + USER_AUTHENTICATION_USER_ID_COLUMN + ", " + 
> USER_AUTHENTICATION_AUTHENTICATION_TYPE_COLUMN + ", " + 
> USER_AUTHENTICATION_AUTHENTICATION_KEY_COLUMN + ", " + 
> USER_AUTHENTICATION_CREATE_TIME_COLUMN + ", " + 
> USER_AUTHENTICATION_UPDATE_TIME_COLUMN + ")" +
>...
>   );
> }
> ```
> 
> Breaking this apart into several queries and a set of insert statements 
> might help; but I thought that it would be difficult since the updated 
> UserEntity class would be out of sync with the original table. What do others 
> do in this situation?  Maybe if I need to go this route, I would create a new 
> `user` table rather than upgrade the `users` table?  In the end, I would drop 
> the old `user` table, but the OldUserEntitiy class will need to live for many 
> versions of Ambari.
> 
> Jonathan Hurley wrote:
> My only concern here was being able to run this again after it fails. 
> Should we just drop the new table in this method if it exists? Then we can 
> keep this block as-is...

This could be dangerous, depending on where it fails. If it fails after 
removing the data from the Users table (`"delete from " + USERS_TABLE...`), 
then we will loose data when we drop the new table.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64544/#review193564
---


On Dec. 13, 2017, 5:01 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64544/
> ---
> 
> (Updated Dec. 13, 2017, 5:01 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Molnar.
> 
> 
> Bugs: AMBARI-22577
> https://issues.apache.org/jira/browse/AMBARI-22577
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Migrate data from the users table (pre-Ambari 3.0.0) to the updated users 
> table and user_authentication tables.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessor.java 
> 549c0fd7e8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/UserAuthenticationDAO.java
>  513e78200d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserAuthenticationEntity.java
>  27514f648c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProvider.java
>  20a06ccd54 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
>  2de60957bc 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e58a04e4b2 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql cc589e47f2 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 7cd083db4c 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql ff232ebe3e 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 
> d7c09f3381 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b4929b308d 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  29f9d917e8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderForDNWithSpaceTest.java
>  10076b0876 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderTest.java
>  43d4d6b0c0 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog300Test.java
>  747f99b618 
> 
> 
> Diff: https://reviews.apache.org/r/64544/diff/2/
> 
> 
> Testing
> ---
> 
> manually tested
> 
> # Local test results: 
> ```
> [INFO] 
>

Review Request 64707: AMBARI-22669 Ranger stack script changes to fix missing directory failure for blueprint installation

2017-12-19 Thread Mugdha Varadkar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64707/
---

Review request for Ambari, Gautam Borad, Jonathan Hurley, and Vishal Suvagia.


Bugs: AMBARI-22669
https://issues.apache.org/jira/browse/AMBARI-22669


Repository: ambari


Description
---

Updating stack scripts code to remove use of commandParams/version to build 
system path. For upgrade, using upgrade_summary module to get upgrade-to 
version ie (target version).


Diffs
-

  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
 5731e6c 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
 d0a725a 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
 9b1f6e2 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py
 bbc438b 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py
 0b37489 
  ambari-server/src/test/python/stacks/2.6/RANGER/test_ranger_admin.py e8bacbd 


Diff: https://reviews.apache.org/r/64707/diff/1/


Testing
---

Tested Ranger Installation.

Ran 255 tests in 7.877s

OK
--
Total run:1210
Total errors:0
Total failures:0
OK


Thanks,

Mugdha Varadkar