----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64594/ -----------------------------------------------------------
(Updated Dec. 14, 2017, 9:02 p.m.) Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan. Changes ------- Updated code as per sugegstions. Bugs: AMBARI-22649 https://issues.apache.org/jira/browse/AMBARI-22649 Repository: ambari Description ------- Background : AMBARI-22198 added "stack settings", and AMBARI-22196 introduced "cluster settings" in Ambari. **==========================================================================================** **Library for querying _clusterSettings_ and _stackSettings_ for its contents in command*.json.** **==========================================================================================** One should be able to query for a given **clusterSettings** or **stackSettings**: - by passing in the setting name(one or more) in order to get it back as key-value map, or - just get the value back for a passed-in setting. **Functions for clusterSettings:** **------------------------------** - **get_cluster_setting_entries(setting_names)** : -- Retrieves the passed-in cluster setting entr(y/ies) and their values as a map. If 'setting_names' is passed-in as None : all the settings names and their corresponding values will be returned as map. If 'setting_names' is passed-in as empty set : None will be returned. - **get_cluster_setting_value(setting_name)** : -- Retrieves the passed-in cluster setting entry's value. - **is_security_enabled()** : -- Retrieves the cluster's security status. **Functions for stackSettings:** **----------------------------** Stack settings as of now has 5 settings : stack_name, stack_root, stack_features, stack_tools, stack_packages. stack_name, stack_root have string as values, whereas stack_features, stack_tools, stack_packages have values as JSON. Further there already exists python functions in files : **stack_features.py**, **stack_tools.py** and **stack_select.py**. - **get_stack_setting_entries(setting_names)** : -- Retrieves the passed-in stack setting entr(y/ies) and their values as a map. If 'setting_names' is passed-in as None, all the settings names and their corresponding values will be returned as map. If 'setting_names' is passed-in as empty set : None will be returned. - **get_stack_setting_value(setting_name)**: -- Retrieves the passed-in stack setting entry's value. - **get_stack_name()**: -- Retrieves the stack name. - **get_stack_root()**: -- Retrieves the stack root. **Modifications in _stack_features.py, stack_tools.py and stack_select.py_ files:** **-----------------------------------------------------------------------------** - Given that these already exist and as of now they read the relevant stack setting from *configurations/cluster_env*. - Thus, code has been added to try reading from /stackSettings first by calling the new fn.() get_stack_setting_value(). if setting not found, go for the fall back *configurations/cluster_env* (which would be removed soon, when we remove cluster_env). Diffs (updated) ----- ambari-common/src/main/python/resource_management/libraries/functions/cluster_settings.py PRE-CREATION ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py 92823b0 ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py b741a33 ambari-common/src/main/python/resource_management/libraries/functions/stack_settings.py PRE-CREATION ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py d9233a3 Diff: https://reviews.apache.org/r/64594/diff/2/ Changes: https://reviews.apache.org/r/64594/diff/1-2/ Testing (updated) ------- Python UT passes. **Testing:** Tested on live cluster **=================** **clusterSettings:** **=================** **A.** get_cluster_setting_entries(): **----------------------------------** - 1. Retrieve **single** setting : 'recovery_enabled' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled']) **o/p**: {'recovery_enabled': True} - 2. Retrieve **two** settings : 'recovery_enabled', 'sysprep_skip_setup_jce' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce']) **o/p**: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 3. Retrieve settings where passed in empty set -> Expected nothing returned -- In get_cluster_setting_entries(). Passed-in setting(s) : set([]) **o/p**: None - 4. Retrieve **three** settings : 'smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce' -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['smokeuser', 'recovery_enabled', 'sysprep_skip_setup_jce']) **o/p**: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False, 'smokeuser': 'ambari-qa'} - 5. Retrieve **three** settings where **middle setting is non-existent** -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'abc', 'sysprep_skip_setup_jce']) **o/p** : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 6. Retrieve **three** settings where **1st setting is non-existent** -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['abc', 'recovery_enabled', 'sysprep_skip_setup_jce']) **o/p** : {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 7. Retrieve **three** settings where **last setting is non-existent** -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['recovery_enabled', 'sysprep_skip_setup_jce', 'abc']) **o/p**: {'recovery_enabled': True, 'sysprep_skip_setup_jce': False} - 8. Retrieve passed in setting which is **non-existent** -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1']) **o/p** : None - 9. Retrieve **two** passed in settings and both are non-existent -- In get_cluster_setting_entries(). Passed-in setting(s) : set(['non-existent1', 'non-existent2']) **o/p** : None - 10. Retrieve settings where set passed in is None. -> **returns all settings.** -- In get_cluster_setting_entries(). Passed-in setting(s) : None **o/p**: {'security_enabled': 'false', 'namenode_rolling_restart_timeout': '4200', 'enable_external_ranger': 'false', 'override_uid': 'true', 'kerberos_domain': 'EXAMPLE.COM', 'one_dir_per_partition': 'false', 'agent_mounts_ignore_list': '', 'repo_ubuntu_template': '{{package_type}} {{base_url}} {{components}}', 'ignore_groupsusers_create': 'false', 'alerts_repeat_tolerance': '1', 'hide_yarn_memory_widget': 'false', 'fetch_nonlocal_groups': 'true', 'manage_dirs_on_root': 'true', 'recovery_lifetime_max_count': '1024', 'recovery_type': 'AUTO_START', 'ignore_bad_mounts': 'false', 'recovery_window_in_minutes': '60', 'sysprep_skip_copy_tarballs_hdfs': 'false', 'user_group': 'hadoop', 'namenode_rolling_restart_safemode_exit_timeout': '3600', 'recovery_retry_interval': '5', 'sysprep_skip_copy_oozie_share_lib_to_hdfs': 'false', 'sysprep_skip_setup_jce': 'false', 'manage_hive_fsroot': 'true', 'service_check_type': 'full', 'recovery_enabled': 'true', 'recovery_max_count': '6', 'sys prep_skip_create_users_and_groups': 'false', 'smokeuser_keytab': '/etc/security/keytabs/smokeuser.headless.keytab', 'managed_hdfs_resource_property_names': 'false', 'smokeuser': 'ambari-qa', 'sysprep_skip_copy_fast_jar_hdfs': 'false'} - 11. Retrieve settings. Passed is in List (Expected Set) -- In get_cluster_setting_entries(). Passed-in setting(s) : ['recovery_enabled'] 'setting_names' type expected to be a set. Passed-in type is : <type 'list'> **o/p**: None **B.** get_cluster_setting_value(): **--------------------------------** - 1. Retrieve cluster_setting : 'hide_yarn_memory_widget' -- In get_cluster_setting_value(). Passed-in setting : hide_yarn_memory_widget **o/p** : false - 2. Retrieve cluster_setting : 'recovery_max_count' -- In get_cluster_setting_value(). Passed-in setting : recovery_max_count **o/p** : 6 - 3. Retrieve cluster_setting where passed in setting is 'non_existing' --> Empty string -- In get_cluster_setting_value(). Passed-in setting : non_existing **o/p** : None - 4. Retrieve cluster_setting setting passed in is "None" -- In get_cluster_setting_value(). Passed-in setting : None **o/p** : None **C**. is_security_enabled(): **-------------------------** - 1. Retrieve security state of cluster by calling 'is_security_enabled()' which in turn calls **get_cluster_setting_value()** -- In get_cluster_setting_value(). Passed-in setting : security_enabled **o/p** : false **=================** **stackSettings**: **=================** **A.** Tests for **get_stack_setting_entries()** **--------------------------------------------** - 1. Retrieve : **stack_name** -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name']) o/p: {'stack_name': 'HDP'} - 2. Retrieve : **stack_root** -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_root']) **o/p**: {'stack_root': '{"HDP":"/usr/hdp"}'} - 3. Retrieve **three** stack settings: **'stack_name', 'stack_root', 'stack_tools'** -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools']) **o/p** : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'} - 4. Retrieve **four** stack settings : **'BAD_VALUE', 'stack_name', 'stack_root', 'stack_tools'** ---> (**BAD_VALUE** should get ignored) -- In get_stack_setting_entries(). Passed-in setting(s) : set(['stack_name', 'stack_root', 'stack_tools']) **o/p** : {'stack_name': 'HDP', 'stack_root': '{"HDP":"/usr/hdp"}', 'stack_tools': '{\n "HDP": {\n "stack_selector": [\n "hdp-select",\n "/usr/bin/hdp-select",\n "hdp-select"\n ],\n "conf_selector": [\n "conf-select",\n "/usr/bin/conf-select",\n "conf-select"\n ]\n }\n}'} - 5. Retrieve settings. Passed is in List (Expected Set) -- In get_stack_setting_entries(). Passed-in setting(s) : ['stack_tools'] 'setting_names' type expected to be a set. Passed-in type is : <type 'list'> **o/p**: None **B.** Tests for existing and modified : **stack_select.get_packages()** which internally now calls **stack_settings.get_stack_setting_value()** **------------------------------------------------------------------------------------------** - 1. Retrieve **specific component package** with call -> stack_select.get_packages(orchestration, "HDFS", "DATANODE") -- In get_stack_setting_value(). Passed-in setting : stack_packages **o/p** : ['hadoop-hdfs-datanode'] - 2. Retrieve **specific component package** with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "ZOOKEEPER_CLIENT") -- In get_stack_setting_value(). Passed-in setting : stack_packages **o/p** : ['zookeeper-client'] - 3. Retrieve **non-existing component package** with call -> stack_select.get_packages(orchestration, "ZOOKEEPER", "NON_EXISTING") -- In get_stack_setting_value(). Passed-in setting : stack_packages Skipping stack-select on ABC because it does not exist in the stack-select package structure. **o/p** : None **C**. Tests for existing and modified : **stack_features.check_stack_feature()** which internally now calls **stack_settings.get_stack_setting_value()** **------------------------------------------------------------------------------------------** - 1. Retrieve **specific component stack feature** with call -> stack_features.check_stack_feature("hive_server_interactive", "2.6") -- In get_stack_setting_value(). Passed-in setting : stack_features **o/p** : True - 2. Retrieve **specific component stack feature** with call -> stack_features.check_stack_feature("hive_server_interactive", "2.2") -- In get_stack_setting_value(). Passed-in setting : stack_features **o/p** : False **D**. Test for existing and modified : **stack_tools.get_stack_tool()** which internally now calls **stack_settings.get_stack_setting_value()** **-------------------------------------------------------------------------------------------** - 1. Retrieve **stack relevant tools** with call -> stack_tools.get_stack_tool("stack_selector") -- In get_stack_setting_value(). Passed-in setting : stack_tools **o/p**: ('hdp-select', '/usr/bin/hdp-select', 'hdp-select') Thanks, Swapan Shridhar