-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50731/
-----------------------------------------------------------

(Updated Aug. 3, 2016, 5:21 p.m.)


Review request for Ambari, Alexander Denissov and Lav Jain.


Changes
-------

Review Feedback


Bugs: AMBARI-17996
    https://issues.apache.org/jira/browse/AMBARI-17996


Repository: ambari


Description
-------

HAWQ service advisor shows wrong recommendations on edge cases


Diffs (updated)
-----

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 0d6f2b2 
  
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 ab43898 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 e8b0f15 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
 39ebdaf 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
 fb6cb97 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 6c0fb70 
  
ambari-server/src/main/java/org/apache/ambari/server/view/ViewDataMigrationUtility.java
 2906100 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
 9efa4f9 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 7e9c0f6 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 7064de3 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/atlas-solrconfig.xml
 cba4a4e 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 9baaa1c 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
 529c492 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/utils.py
 95c3dcc 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
0777dc6 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-audit.xml
 b4c0790 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-plugin-properties.xml
 3949402 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-policymgr-ssl.xml
 cf4a82e 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-security.xml
 91061d1 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/configuration/ranger-knox-plugin-properties.xml
 ae9314b 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
 44dc61a 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
 ecb103c 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 eded4bd 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 50c1555 
  
ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
 865aa73 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-audit.xml
 4dc51eb 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-policymgr-ssl.xml
 b1f6e1e 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-security.xml
 983702f 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-site.xml
 e504c2d 
  
ambari-server/src/main/resources/common-services/STORM/0.9.3/configuration/ranger-storm-plugin-properties.xml
 faeac96 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 
f80fa8a 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
 bc8947a 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 b2c635d 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
 93c3aaa 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 da577ac 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml
 97283e0 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml
 b4fb928 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/ranger-hbase-plugin-properties.xml
 960c751 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml
 c57c5f0 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/ranger-hive-plugin-properties.xml
 830c539 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/configuration/ranger-knox-plugin-properties.xml
 d5880dd 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-audit.xml
 85c16c8 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-policymgr-ssl.xml
 c761b26 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-security.xml
 1deb9e5 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml
 89c26a5 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-audit.xml
 b0131e7 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-policymgr-ssl.xml
 71ba3a6 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-security.xml
 cfd8a4f 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-audit.xml
 b210fca 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-policymgr-ssl.xml
 a538843 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-security.xml
 ce4074a 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/configuration/ranger-kafka-policymgr-ssl.xml
 24fd407 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/configuration/ranger-knox-audit.xml
 1f3c1d1 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/configuration/ranger-knox-policymgr-ssl.xml
 bb0878f 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/configuration/ranger-knox-security.xml
 9bd1079 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/configuration/ranger-storm-policymgr-ssl.xml
 5672f04 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/configuration/ranger-storm-security.xml
 f3d7530 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/ranger-yarn-audit.xml
 a6b1baa 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/ranger-yarn-plugin-properties.xml
 97867cc 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/ranger-yarn-policymgr-ssl.xml
 5410104 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/ranger-yarn-security.xml
 5f69962 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml
 68f0e04 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
7661205 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 dbdecc7 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 4c09f43 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
6ba3d32 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
cf0be76 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 0f5c126 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 0f5c126 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
 36677a1 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-plugin-properties.xml
 fd623cb 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-policymgr-ssl.xml
 dcffb63 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-security.xml
 ea0a026 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/FALCON/configuration/falcon-atlas-application.properties.xml
 e653651 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml
 77f7be2 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
 14ab470 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
37a7e82 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
 6173275 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/ranger-storm-policymgr-ssl.xml
 5672f04 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/ranger-storm-security.xml
 f3d7530 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
 90b2243 
  ambari-server/src/main/resources/stacks/stack_advisor.py 437fe4f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 3c97ce9 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
 9463749 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
 874da31 
  ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
7ececa0 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
186ff72 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 0921edc 
  ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
e8e9320 
  
ambari-server/src/test/resources/kerberos/test_get_referenced_identity_descriptor.json
 0ad71d3 
  ambari-web/app/controllers/main/service/item.js 6ce811d 
  ambari-web/app/messages.js 1f42685 
  ambari-web/app/models/host_component.js dca7ff0 
  ambari-web/app/views/main/service/item.js a007e17 
  ambari-web/app/views/main/service/services/hdfs.js 40fb761 
  ambari-web/test/views/main/service/item_test.js f318185 
  contrib/views/hive-next/pom.xml 3cd4c4a 
  
contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/DataMigrator.java
 b33df65 
  contrib/views/hive-next/src/main/resources/view.xml 1754686 
  contrib/views/hive/pom.xml b26a070 
  contrib/views/slider/pom.xml 0ddcb84 
  contrib/views/tez/pom.xml 26b0c49 
  contrib/views/zeppelin/pom.xml 52efdce 

Diff: https://reviews.apache.org/r/50731/diff/


Testing
-------

Manually tested on live cluster.

Added unit tests
```
test_hawq_master_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_hawq_segment_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_hawq_standby_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_missing_configs (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_exception_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
handlers could be found for logger "ambari_alerts"
ok
test_missing_configs 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_missing_slave_file 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_successful_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_empty_db_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_registration_status_plural 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
test_install_default (test_hawqmaster.TestHawqMaster) ... ok
test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
Run custom command Remove HAWQ Standby ... 2016-08-02 18:01:23,293 - Removing 
HAWQ Standby Master ...
ok
test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
Run custom command Resync HAWQ Standby ... 2016-08-02 18:01:23,295 - HAWQ 
Standby Master Re-Sync started in fast mode...
ok
test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
Kerberized ... 2016-08-02 18:01:23,298 - Executing HAWQ Check ...
ok
test_run_hawq_check_case10 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 10: HDFS-HA, YARN Resource Management Non YARN_HA, 
Kerberized ... 2016-08-02 18:01:23,304 - Executing HAWQ Check ...
ok
test_run_hawq_check_case11 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 11: HDFS-HA, YARN Resource Management YARN_HA, Not 
Kerberized ... 2016-08-02 18:01:23,309 - Executing HAWQ Check ...
ok
test_run_hawq_check_case12 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 12: HDFS-HA, YARN Resource Management YARN_HA, 
Kerberized ... 2016-08-02 18:01:23,315 - Executing HAWQ Check ...
ok
test_run_hawq_check_case2 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 2: Non HDFS-HA, Standalone Resource Management, 
Kerberized ... 2016-08-02 18:01:23,320 - Executing HAWQ Check ...
ok
test_run_hawq_check_case3 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 3: Non HDFS-HA, YARN Resource Management Non YARN_HA, 
Not Kerberized ... 2016-08-02 18:01:23,327 - Executing HAWQ Check ...
ok
test_run_hawq_check_case4 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 4: Non HDFS-HA, YARN Resource Management Non YARN_HA, 
Kerberized ... 2016-08-02 18:01:23,332 - Executing HAWQ Check ...
ok
test_run_hawq_check_case5 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 5: Non HDFS-HA, YARN Resource Management YARN_HA, Not 
Kerberized ... 2016-08-02 18:01:23,337 - Executing HAWQ Check ...
ok
test_run_hawq_check_case6 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 6: Non HDFS-HA, YARN Resource Management YARN_HA, 
Kerberized ... 2016-08-02 18:01:23,343 - Executing HAWQ Check ...
ok
test_run_hawq_check_case7 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 7: HDFS-HA, Standalone Resource Management, Not 
Kerberized ... 2016-08-02 18:01:23,348 - Executing HAWQ Check ...
ok
test_run_hawq_check_case8 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 8: HDFS-HA, Standalone Resource Management, Kerberized 
... 2016-08-02 18:01:23,355 - Executing HAWQ Check ...
ok
test_run_hawq_check_case9 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 9: HDFS-HA, YARN Resource Management Non YARN_HA, Not 
Kerberized ... 2016-08-02 18:01:23,360 - Executing HAWQ Check ...
ok
test_start_default (test_hawqmaster.TestHawqMaster) ... 2016-08-02 18:01:23,368 
- Skipping ssh key exchange with HAWQ hosts as hawq_ssh_exkeys is either set to 
false or is not available in hawq-env.xml
ok
test_stop_cluster_immediate (test_hawqmaster.TestHawqMaster)
Run Stop HAWQ Cluster Immediate Mode ... ok
test_stop_default (test_hawqmaster.TestHawqMaster)
Run Stop HAWQMASTER ... ok
test_configure_default (test_hawqsegment.TestHawqSegment) ... ok
test_install_default (test_hawqsegment.TestHawqSegment) ... ok
test_start_default (test_hawqsegment.TestHawqSegment) ... ok
test_stop_default (test_hawqsegment.TestHawqSegment) ... ok
test_activate_hawq_standby (test_hawqstandby.TestHawqStandby)
Test Activate HAWQ Standby Command ... 2016-08-02 18:01:23,395 - Activating 
HAWQ standby...
ok
test_configure_default (test_hawqstandby.TestHawqStandby) ... ok
test_install_default (test_hawqstandby.TestHawqStandby) ... ok
test_start_default (test_hawqstandby.TestHawqStandby) ... ok
test_stop_default (test_hawqstandby.TestHawqStandby) ... ok
test_createComponentLayoutRecommendations_hawq_1_Host 
(test_service_advisor.TestHAWQ200ServiceAdvisor) ... ok
test_createComponentLayoutRecommendations_hawq_3_Hosts 
(test_service_advisor.TestHAWQ200ServiceAdvisor)
Test that HAWQSTANDBY is recommended on a 3-node cluster ... ok
test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed
 (test_service_advisor.TestHAWQ200ServiceAdvisor)
Test that HAWQSEGMENT does not get recommended during Add Service Wizard, when 
HAWQ has already been installed ... ok
test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed
 (test_service_advisor.TestHAWQ200ServiceAdvisor)
Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, 
when HAWQ is selected for installation ... ok
test_createComponentLayoutRecommendations_hawqsegment_cluster_install 
(test_service_advisor.TestHAWQ200ServiceAdvisor)
Test that HAWQSEGMENT gets recommended correctly during Cluster Install Wizard, 
when HAWQ is selected for installation ... ok
test_createComponentLayoutRecommendations_no_hawq_3_Hosts 
(test_service_advisor.TestHAWQ200ServiceAdvisor)
Test no failures when there are no HAWQ components ... ok
test_getComponentLayoutValidations_hawq_3_Hosts 
(test_service_advisor.TestHAWQ200ServiceAdvisor)
Test layout validations for HAWQ components on a 3-node cluster ... ok
test_getComponentLayoutValidations_hawqsegment_not_co_located_with_datanode 
(test_service_advisor.TestHAWQ200ServiceAdvisor)
Test validation warning for HAWQ segment not colocated with DATANODE ... ok
test_getComponentLayoutValidations_nohawq_3_Hosts 
(test_service_advisor.TestHAWQ200ServiceAdvisor)
Test no failures when there are no HAWQ components on a 3-node cluster ... ok
test_getHostsForMasterComponent 
(test_service_advisor.TestHAWQ200ServiceAdvisor) ... ok
test_getServiceConfigurationRecommendations 
(test_service_advisor.TestHAWQ200ServiceAdvisor) ... ok
test_validateHAWQHdfsClientConfigurations 
(test_service_advisor.TestHAWQ200ServiceAdvisor) ... ok
test_validateHAWQSiteConfigurations 
(test_service_advisor.TestHAWQ200ServiceAdvisor) ... ok
test_validateHDFSSiteConfigurations 
(test_service_advisor.TestHAWQ200ServiceAdvisor) ... ok
test_generate_hawq_process_status_cmd (test_utils.TestUtils) ... ok

----------------------------------------------------------------------
Ran 65 tests in 0.219s

OK
ServiceAdvisor implementation for service HAWQ was loaded
ServiceAdvisor implementation for service HAWQ was loaded
ServiceAdvisor implementation for service HAWQ was loaded
ServiceAdvisor implementation for service HAWQ was loaded
ServiceAdvisor implementation for service HAWQ was loaded
ServiceAdvisor implementation for service HAWQ was loaded
ServiceAdvisor implementation for service HAWQ was loaded

```


Thanks,

Matt

Reply via email to