Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
/resources/stacks_with_extensions/HDP/0.1/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
> On June 3, 2016, 6:29 p.m., Alejandro Fernandez wrote: > > A lof of files have changed, please give me 24 hours to go over this. Thank > > you The patch really hasn't changed all that much in the past week or so. I did add the Upgrade240 changes to add the 2 new DB tables when upgrading to 2.4 during the last week. Other than that it is just redoing the patch so that it can work over the latest trunk changes. I will be doing a demo of this on June 7th @11am PST. Contact Jayush if you want to be included in the meeting invite. Thanks - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47656/#review136089 --- On June 3, 2016, 4:54 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47656/ > --- > > (Updated June 3, 2016, 4:54 p.m.) > > > Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh > chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth > Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-12885 > https://issues.apache.org/jira/browse/AMBARI-12885 > > > Repository: ambari > > > Description > --- > > The purpose of this proposal is to facilitate adding custom services to an > existing stack. Ideally this would support adding and upgrading custom > services separately from the core services defined in the stack. In > particular we are looking at custom services that need to support several > different stacks (different distributions of Ambari). The release cycle of > the custom services may be different from that of the core stack; that is, a > custom service may be upgraded at a different rate than the core distribution > itself and may be upgraded multiple times within the lifespan of a single > release of the core distribution. > > One possible approach to handling this would be dynamically extending a stack > (after install time). It would be best to extend the stack in packages where > a stack extension package can have one or more custom services. > > > Diffs > - > > ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py > fc1b72a > ambari-agent/src/main/python/ambari_agent/FileCache.py b7c5dee > ambari-server/conf/unix/ambari.properties a88a025 > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java > cf7c391 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > f0928cf > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java > 557ce98 > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java > 2b7fca0 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java > 9f221d5 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > fe9204d > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java > 3721113 >
Re: Review Request 48229: Refactor service_advisor apis to remove passing of stack_advisor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48229/#review136240 --- ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py (line 40) <https://reviews.apache.org/r/48229/#comment201265> This should be removed in the final code ambari-server/src/main/resources/stacks/service_advisor.py (line 51) <https://reviews.apache.org/r/48229/#comment201266> I originally had implemented the service advisor like this. I had modified it specifically NOT to inherit from the DefaultStackAdvisor after comments suggesting that the inheritance didn't make sense. The way I see it both the stack advisor and service advisor should inherit the core functions from a generic advisor but this implementation is the easiest. ambari-server/src/main/resources/stacks/stack_advisor.py (line 412) <https://reviews.apache.org/r/48229/#comment201267> This is really not the method, it is the advisor. ambari-server/src/main/resources/stacks/stack_advisor.py (line 434) <https://reviews.apache.org/r/48229/#comment201268> This is really the advisor not the method. - Tim Thorpe On June 5, 2016, 10:18 p.m., Lav Jain wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48229/ > --- > > (Updated June 5, 2016, 10:18 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Matt, and Tim Thorpe. > > > Bugs: AMBARI-16663 > https://issues.apache.org/jira/browse/AMBARI-16663 > > > Repository: ambari > > > Description > --- > > The original intent was to move the common functions that are used by both > stack and service advisors to a separate utils file to avoid passing > stack_advisor class into service advisor apis. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > 28eb82f > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > 9b34171 > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 1dedf3d > ambari-server/src/main/resources/stacks/service_advisor.py 3d6c293 > ambari-server/src/main/resources/stacks/stack_advisor.py beba225 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py > 5750938 > ambari-server/src/test/python/stacks/2.3/PXF/test_service_advisor.py > c3a63cc > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 424a386 > > Diff: https://reviews.apache.org/r/48229/diff/ > > > Testing > --- > > Lavs-MacBook-Pro:common ljain$ python -m discover -v > test_createComponentLayoutRecommendations_hawq_1_Host > (test_stack_advisor.TestHDP23StackAdvisor) ... ServiceAdvisor implementation > for service HAWQ was loaded > [u'c6401.ambari.apache.org'] > [u'c6401.ambari.apache.org'] > ok > test_createComponentLayoutRecommendations_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSTANDBY is recommended on a 3-node cluster ... ServiceAdvisor > implementation for service HAWQ was loaded > [u'c6401.ambari.apache.org', u'c6402.ambari.apache.org', > u'c6403.ambari.apache.org'] > [u'c6401.ambari.apache.org', u'c6402.ambari.apache.org', > u'c6403.ambari.apache.org'] > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT does not get recommended during Add Service Wizard, > when HAWQ has already been installed ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, > when HAWQ is selected for installation ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_cluster_install > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Cluster Install > Wizard, when HAWQ is selected for installation ... ServiceAdvisor > implementation for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_no_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test no failures when there are no HAWQ components ... ok > test
Re: Review Request 48234: Falcon server fails to start, HDP 2.4 to use data-mirroring directory, HDP 2.5 to use extensions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48234/#review136238 --- Ship it! Ship It! - Tim Thorpe On June 3, 2016, 10:47 p.m., Alejandro Fernandez wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48234/ > --- > > (Updated June 3, 2016, 10:47 p.m.) > > > Review request for Ambari, Andrew Onischuk, Di Li, Dmitro Lisnichenko, > Jonathan Hurley, Jayush Luniya, Nate Cole, and Tim Thorpe. > > > Bugs: AMBARI-17037 > https://issues.apache.org/jira/browse/AMBARI-17037 > > > Repository: ambari > > > Description > --- > > Fresh installation and EU/RU of Falcon is failing. > In HDP 2.4, Falcon used the data mirroring folder. > In HDP 2.5, data mirroring is no longer used and instead uses the extensions > folder that must be uploaded to HDFS. > > This means that EU/RU from HDP 2.4 to 2.5 must also upload the extensions > folder to HDFS > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > 34d1a1a > > ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py > 4c13aaa > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py > 5e25325 > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py > 7c2cdf4 > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py > 8c2ad8e > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > 8b669e8 > ambari-server/src/test/python/stacks/2.0.6/configs/default.json 8d04e36 > > Diff: https://reviews.apache.org/r/48234/diff/ > > > Testing > --- > > Python unit tests passed. > > -- > Total run:1052 > Total errors:0 > Total failures:0 > OK > > Express Upgrade is currently broken, so will commit this patch and retest > once it is stable again in Hadoop Core. > > > Thanks, > > Alejandro Fernandez > >
Re: Review Request 48229: Refactor service_advisor apis to remove passing of stack_advisor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48229/#review136343 --- Ship it! Ship It! - Tim Thorpe On June 6, 2016, 6:40 p.m., Lav Jain wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48229/ > --- > > (Updated June 6, 2016, 6:40 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Matt, and Tim Thorpe. > > > Bugs: AMBARI-16663 > https://issues.apache.org/jira/browse/AMBARI-16663 > > > Repository: ambari > > > Description > --- > > The original intent was to move the common functions that are used by both > stack and service advisors to a separate utils file to avoid passing > stack_advisor class into service advisor apis. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > 28eb82f > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > 9b34171 > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 1dedf3d > ambari-server/src/main/resources/stacks/service_advisor.py 3d6c293 > ambari-server/src/main/resources/stacks/stack_advisor.py beba225 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py > 5750938 > ambari-server/src/test/python/stacks/2.3/PXF/test_service_advisor.py > c3a63cc > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 424a386 > > Diff: https://reviews.apache.org/r/48229/diff/ > > > Testing > --- > > Lavs-MacBook-Pro:common ljain$ python -m discover -v > test_createComponentLayoutRecommendations_hawq_1_Host > (test_stack_advisor.TestHDP23StackAdvisor) ... ServiceAdvisor implementation > for service HAWQ was loaded > [u'c6401.ambari.apache.org'] > [u'c6401.ambari.apache.org'] > ok > test_createComponentLayoutRecommendations_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSTANDBY is recommended on a 3-node cluster ... ServiceAdvisor > implementation for service HAWQ was loaded > [u'c6401.ambari.apache.org', u'c6402.ambari.apache.org', > u'c6403.ambari.apache.org'] > [u'c6401.ambari.apache.org', u'c6402.ambari.apache.org', > u'c6403.ambari.apache.org'] > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT does not get recommended during Add Service Wizard, > when HAWQ has already been installed ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, > when HAWQ is selected for installation ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_cluster_install > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Cluster Install > Wizard, when HAWQ is selected for installation ... ServiceAdvisor > implementation for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_no_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test no failures when there are no HAWQ components ... ok > test_createComponentLayoutRecommendations_pxf_add_service_wizard_already_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF does not get recommended during Add Service Wizard, when PXF > has already been installed ... ServiceAdvisor implementation for service PXF > was loaded > ok > test_createComponentLayoutRecommendations_pxf_add_service_wizard_to_be_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF gets recommended correctly during Add Service Wizard, when PXF > is selected for installation ... ServiceAdvisor implementation for service > PXF was loaded > ok > test_createComponentLayoutRecommendations_pxf_cluster_install > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF gets recommended correctly during Cluster Install Wizard, when > PXF is selected for installation ... ServiceAdvisor implementation for > service PXF was loaded > ok > test_getComponentLayoutValidations_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdviso
Re: Review Request 48561: Falcon to create data-mirroring directory in HDFS if extensions is supported
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48561/#review137066 --- Ship it! Ship It! - Tim Thorpe On June 10, 2016, 6:13 p.m., Alejandro Fernandez wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48561/ > --- > > (Updated June 10, 2016, 6:13 p.m.) > > > Review request for Ambari, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Jayush > Luniya, Nate Cole, and Tim Thorpe. > > > Bugs: AMBARI-17171 > https://issues.apache.org/jira/browse/AMBARI-17171 > > > Repository: ambari > > > Description > --- > > If Falcon extensions is supported, Ambari still needs to create the > /apps/falcon/data-mirroring directory in HDFS. > Also, RU/EU does not need to explicitly create the extensions directory in > the pre_upgrade_restart function since the start function will create it. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py > b5b3a34 > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py > 0d2cd8e > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py > 41954a5 > > Diff: https://reviews.apache.org/r/48561/diff/ > > > Testing > --- > > Ran python unit tests, > > Total run:1055 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Alejandro Fernandez > >
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
/test/resources/stacks_with_extensions/HDP/0.1/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
> On June 3, 2016, 9:46 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql, line 32 > > <https://reviews.apache.org/r/47656/diff/5/?file=1405816#file1405816line32> > > > > Let's be consistent with the name, UQ_* Just noting that the code was copied and pasted from the stack table which uses unq_stack. Do you want me to change all usages of unq? cat src/main/resources/Ambari-DDL-Derby-CREATE.sql | grep unq CONSTRAINT unq_stack UNIQUE (stack_name, stack_version)); CONSTRAINT unq_extension UNIQUE(extension_name, extension_version)); CONSTRAINT unq_extension_link UNIQUE(stack_id, extension_id)); CONSTRAINT unq_scdesiredstate_name UNIQUE(component_name, service_name, cluster_id), CONSTRAINT unq_remote_ambari_cluster UNIQUE (name)); > On June 3, 2016, 9:46 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java, > > line 3798 > > <https://reviews.apache.org/r/47656/diff/5/?file=1405774#file1405774line3798> > > > > Add javadoc to new methods > > Someone new reading the code should know what an extension is. I added in some comments to all the methods describing what an extension is after I've finished the documentation on the wiki, I will revisit some of these changes and reference the wiki. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47656/#review136105 ------- On June 14, 2016, 7:05 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47656/ > --- > > (Updated June 14, 2016, 7:05 p.m.) > > > Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh > chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth > Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-12885 > https://issues.apache.org/jira/browse/AMBARI-12885 > > > Repository: ambari > > > Description > --- > > The purpose of this proposal is to facilitate adding custom services to an > existing stack. Ideally this would support adding and upgrading custom > services separately from the core services defined in the stack. In > particular we are looking at custom services that need to support several > different stacks (different distributions of Ambari). The release cycle of > the custom services may be different from that of the core stack; that is, a > custom service may be upgraded at a different rate than the core distribution > itself and may be upgraded multiple times within the lifespan of a single > release of the core distribution. > > One possible approach to handling this would be dynamically extending a stack > (after install time). It would be best to extend the stack in packages where > a stack extension package can have one or more custom services. > > > Diffs > - > > ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py > fc1b72a > ambari-agent/src/main/python/ambari_agent/FileCache.py b7c5dee > ambari-server/conf/unix/ambari.properties a88a025 > ambari-server/src/main/assemblies/server.xml 1560d8d > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java > cf7c391 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > f0928cf > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java > 557ce98 > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java > 2b7fca0 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java > b488af3 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementContr
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
/test/resources/stacks_with_extensions/HDP/0.1/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
> On May 23, 2016, 10:49 a.m., Alexander Denissov wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java, > > line 305 > > <https://reviews.apache.org/r/47656/diff/2/?file=1389298#file1389298line305> > > > > a lot of commented out code here I have removed most if not all of the commented code. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47656/#review134330 ------- On May 20, 2016, 7:03 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47656/ > --- > > (Updated May 20, 2016, 7:03 p.m.) > > > Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh > chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth > Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-12885 > https://issues.apache.org/jira/browse/AMBARI-12885 > > > Repository: ambari > > > Description > --- > > The purpose of this proposal is to facilitate adding custom services to an > existing stack. Ideally this would support adding and upgrading custom > services separately from the core services defined in the stack. In > particular we are looking at custom services that need to support several > different stacks (different distributions of Ambari). The release cycle of > the custom services may be different from that of the core stack; that is, a > custom service may be upgraded at a different rate than the core distribution > itself and may be upgraded multiple times within the lifespan of a single > release of the core distribution. > > One possible approach to handling this would be dynamically extending a stack > (after install time). It would be best to extend the stack in packages where > a stack extension package can have one or more custom services. > > > Diffs > - > > ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py > fcea23f > ambari-agent/src/main/python/ambari_agent/FileCache.py b7c5dee > ambari-server/conf/unix/ambari.properties 9f1692e > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java > 9c864b6 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > c54fe3f > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java > 557ce98 > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java > f5e8f39 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java > d6b9d0e > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > dd3d098 > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java > 3721113 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extens
Re: Review Request 47858: Cache service advisors when stack advisor is loaded
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47858/#review135049 --- Ship it! Ship It! - Tim Thorpe On May 26, 2016, 6:50 p.m., Lav Jain wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47858/ > --- > > (Updated May 26, 2016, 6:50 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Matt, and Tim Thorpe. > > > Bugs: AMBARI-16665 > https://issues.apache.org/jira/browse/AMBARI-16665 > > > Repository: ambari > > > Description > --- > > With the current implementation, service advisors for the same service are > loaded multiple times in the stack advisor. > > The service advisors should be cached when the stack advisor is loaded and > used where necessary. > > > Diffs > - > > ambari-server/src/main/resources/stacks/stack_advisor.py 63885d2 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py > 53c5574 > > Diff: https://reviews.apache.org/r/47858/diff/ > > > Testing > --- > > Lavs-MacBook-Pro:common ljain$ python -m discover -v > test_createComponentLayoutRecommendations_hawq_1_Host > (test_stack_advisor.TestHDP23StackAdvisor) ... ServiceAdvisor implementation > for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSTANDBY is recommended on a 3-node cluster ... ServiceAdvisor > implementation for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT does not get recommended during Add Service Wizard, > when HAWQ has already been installed ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, > when HAWQ is selected for installation ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_cluster_install > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Cluster Install > Wizard, when HAWQ is selected for installation ... ServiceAdvisor > implementation for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_no_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test no failures when there are no HAWQ components ... ok > test_createComponentLayoutRecommendations_pxf_add_service_wizard_already_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF does not get recommended during Add Service Wizard, when PXF > has already been installed ... ServiceAdvisor implementation for service PXF > was loaded > ok > test_createComponentLayoutRecommendations_pxf_add_service_wizard_to_be_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF gets recommended correctly during Add Service Wizard, when PXF > is selected for installation ... ServiceAdvisor implementation for service > PXF was loaded > ok > test_createComponentLayoutRecommendations_pxf_cluster_install > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF gets recommended correctly during Cluster Install Wizard, when > PXF is selected for installation ... ServiceAdvisor implementation for > service PXF was loaded > ok > test_getComponentLayoutValidations_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test layout validations for HAWQ components on a 3-node cluster ... > ServiceAdvisor implementation for service HAWQ was loaded > ok > test_getComponentLayoutValidations_hawqsegment_not_co_located_with_datanode > (test_stack_advisor.TestHDP23StackAdvisor) > Test validation warning for HAWQ segment not colocated with DATANODE ... > ServiceAdvisor implementation for service HAWQ was loaded > ok > test_getComponentLayoutValidations_nohawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test no failures when there are no HAWQ components on a 3-node cluster ... ok > test_getComponentLayoutValidations_pxf_co_located_with_nn_and_dn > (test_stack_advisor.TestHDP23StackAdvisor) > Test NO warning is generated when PXF is co-located
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47858: Cache service advisors when stack advisor is loaded
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47858/#review135294 --- Ship it! Ship It! - Tim Thorpe On May 27, 2016, 7:11 p.m., Lav Jain wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47858/ > --- > > (Updated May 27, 2016, 7:11 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Matt, and Tim Thorpe. > > > Bugs: AMBARI-16665 > https://issues.apache.org/jira/browse/AMBARI-16665 > > > Repository: ambari > > > Description > --- > > With the current implementation, service advisors for the same service are > loaded multiple times in the stack advisor. > > The service advisors should be cached when the stack advisor is loaded and > used where necessary. > > > Diffs > - > > ambari-server/src/main/resources/stacks/stack_advisor.py 63885d2 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py > 1d6a85c > > Diff: https://reviews.apache.org/r/47858/diff/ > > > Testing > --- > > Lavs-MacBook-Pro:common ljain$ python -m discover -v > test_createComponentLayoutRecommendations_hawq_1_Host > (test_stack_advisor.TestHDP23StackAdvisor) ... ServiceAdvisor implementation > for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSTANDBY is recommended on a 3-node cluster ... ServiceAdvisor > implementation for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT does not get recommended during Add Service Wizard, > when HAWQ has already been installed ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, > when HAWQ is selected for installation ... ServiceAdvisor implementation for > service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_hawqsegment_cluster_install > (test_stack_advisor.TestHDP23StackAdvisor) > Test that HAWQSEGMENT gets recommended correctly during Cluster Install > Wizard, when HAWQ is selected for installation ... ServiceAdvisor > implementation for service HAWQ was loaded > ok > test_createComponentLayoutRecommendations_no_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test no failures when there are no HAWQ components ... ok > test_createComponentLayoutRecommendations_pxf_add_service_wizard_already_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF does not get recommended during Add Service Wizard, when PXF > has already been installed ... ServiceAdvisor implementation for service PXF > was loaded > ok > test_createComponentLayoutRecommendations_pxf_add_service_wizard_to_be_installed > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF gets recommended correctly during Add Service Wizard, when PXF > is selected for installation ... ServiceAdvisor implementation for service > PXF was loaded > ok > test_createComponentLayoutRecommendations_pxf_cluster_install > (test_stack_advisor.TestHDP23StackAdvisor) > Test that PXF gets recommended correctly during Cluster Install Wizard, when > PXF is selected for installation ... ServiceAdvisor implementation for > service PXF was loaded > ok > test_getComponentLayoutValidations_hawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test layout validations for HAWQ components on a 3-node cluster ... > ServiceAdvisor implementation for service HAWQ was loaded > ok > test_getComponentLayoutValidations_hawqsegment_not_co_located_with_datanode > (test_stack_advisor.TestHDP23StackAdvisor) > Test validation warning for HAWQ segment not colocated with DATANODE ... > ServiceAdvisor implementation for service HAWQ was loaded > ok > test_getComponentLayoutValidations_nohawq_3_Hosts > (test_stack_advisor.TestHDP23StackAdvisor) > Test no failures when there are no HAWQ components on a 3-node cluster ... ok > test_getComponentLayoutValidations_pxf_co_located_with_nn_and_dn > (test_stack_advisor.TestHDP23StackAdvisor) > Test NO warning is generated when PXF is co-located
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.1/services/PIG/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HBASE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/global.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hbase-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hdfs-site.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/package/dummy-script.py PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HIVE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/MAPREDUCE/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/ZOOKEEPER/metainfo.xml PRE-CREATION Diff: https://reviews.apache.org/r/47656/diff/ Testing --- Successfully ran all the following tests in ambari/ambari-server/src/test/java/org/apache/ambari/server/stack src/test/java/org/apache/ambari/server/api/services/ExtensionsServiceTest.java src/test/java/org/apache/ambari/server/controller/internal/ExtensionResourceProviderTest.java src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java src/test/java/org/apache/ambari/server/stack/KerberosDescriptorTest.java src/test/java/org/apache/ambari/server/stack/QuickLinksConfigurationModuleTest.java src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java src/test/java/org/apache/ambari/server/stack/StackManagerExtensionTest.java src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java src/test/java/org/apache/ambari/server/stack/StackManagerTest.java src/test/java/org/apache/ambari/server/stack/ThemeModuleTest.java Thanks, Tim Thorpe
Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services
> On June 15, 2016, 8:21 p.m., Alejandro Fernandez wrote: > > Tim, let me know if you want me to commit this to trunk. Can you commit the latest patch that fixes the UpgradeCatalog240Test issue both to trunk and 2.4? Thanks - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47656/#review137828 --- On June 20, 2016, 4:18 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47656/ > --- > > (Updated June 20, 2016, 4:18 p.m.) > > > Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh > chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth > Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-12885 > https://issues.apache.org/jira/browse/AMBARI-12885 > > > Repository: ambari > > > Description > --- > > The purpose of this proposal is to facilitate adding custom services to an > existing stack. Ideally this would support adding and upgrading custom > services separately from the core services defined in the stack. In > particular we are looking at custom services that need to support several > different stacks (different distributions of Ambari). The release cycle of > the custom services may be different from that of the core stack; that is, a > custom service may be upgraded at a different rate than the core distribution > itself and may be upgraded multiple times within the lifespan of a single > release of the core distribution. > > One possible approach to handling this would be dynamically extending a stack > (after install time). It would be best to extend the stack in packages where > a stack extension package can have one or more custom services. > > > Diffs > - > > ambari-server/conf/unix/ambari.properties a88a025 > ambari-server/src/main/assemblies/server.xml 1560d8d > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java > cf7c391 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > f0928cf > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java > 557ce98 > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java > 2b7fca0 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java > b488af3 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > b5c6fc2 > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionRequest.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionResponse.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java > 3721113 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionLinkResourceProvider.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionResourceProvider.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/se
Re: Review Request 48805: AMBARI-17280. RU to write out client configs that are dependencies of Hive, ATS, and Oozie during upgrades that change configs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48805/#review138056 --- Ship it! Ship It! - Tim Thorpe On June 16, 2016, 5:47 p.m., Alejandro Fernandez wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48805/ > --- > > (Updated June 16, 2016, 5:47 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, > Jonathan Hurley, Nate Cole, and Tim Thorpe. > > > Bugs: AMBARI-17280 > https://issues.apache.org/jira/browse/AMBARI-17280 > > > Repository: ambari > > > Description > --- > > During RU, HiveServer2 is restarted but the newer tez configs have not yet > been saved, which is incorrect because Hive has a dependency on Tez. > This is important when configs change during a major stack upgrade, e.g., HDP > 2.4 -> 2.5. What happens today is, > > * Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from > /etc/tez/2.4.0.0-1/0/ to the new folder > * If configs change during RU, then Hive is restarted and the classpath means > that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 > folder > > > This problem exists for all of these components: > > * HiveServer: depends on Tez and MapReduce clients > * ATS: depends on Tez and Spark clients > * Oozie: depends on Tez, Spark, and MapReduce clients > > This problem only exists when configs change (so crossing major stack > version) and during RU (because it is allowed to change configs during the > middle of restarting services). > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java > fb3ae69 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java > 80bb26c > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py > 63c72f7 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py > ef41453 > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py > 44239c7 > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py > 67466e3 > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py > c79d63b > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py > db22004 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py > 90f885a > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py > d1ec15b > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 4187d64 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 3461ad4 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 426b452 > ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py e53eb4b > > Diff: https://reviews.apache.org/r/48805/diff/ > > > Testing > --- > > Verified during RU from HDP 2.4 to 2.5 with ATS, Hive, Tez, Oozie, and Spark > > Python unit tests passed, > -- > Total run:1062 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Alejandro Fernandez > >
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On March 29, 2016, 12:49 p.m., Nate Cole wrote: > > I think you need a more concrete way of ordering here. What if two > > services are marked as YARN? Which one takes precedence? You may > > want to introduce an in order to > > specifically state how it happens. Order would be a float, such that if we > > have a 1.0 and a 2.0, and some service goes in between, then we don't need > > to reorder the world (just use 1.5, or 1.1 or whatever). > > > > In addition, if you are making separate files, it's conceivable that config > > changes for services can go directly in their respective files. We broke > > them up to separate church-and-state, if you will. > > > > It seems like a more achievable goal might be to allow services to announce > > how they fit into how the upgrade packs work today, without rewriting how > > the whole system works. > > Jayush Luniya wrote: > RE: It seems like a more achievable goal might be to allow services to > announce how they fit into how the upgrade packs work today, without > rewriting how the whole system works. > +1 on allowing add-on/custom service to extend stack upgrade packs > instead of breaking down the entire upgrade pack to service level. > > Tim Thorpe wrote: > We decided we wouldn't split the upgrade xml into pieces for all the > stack services. Instead the goal will be limited to allow custom services to > integrate into the upgrade process by providing their own service upgrade xml > which will define their prereqs, order and process. This means we still need > a way for a service to indicate when during the process each of its steps > needs to be done. > > I am a bit leary of the order number because there are actually two ways > that service steps need to get added. (In the example below I will still use > core services to demonstrate). > > The first case is where a service needs to get integrated into an exists > step. In the example below, a service (HBASE) needs to be backed up prior to > upgrade: > > > KNOX > > > scripts/hbase_upgrade.py > take_snapshot > > > > > The second case is where a service has a new step which just needs to be > included in the order. In the example below, a service (KNOX) needs a new > step added: > > > KAFKA > true > > KNOX_GATEWAY > > > > Nate proposed using an order number but this gets messy in the case of > integrating into an existing step. There would be 2 order numbers, one for > the group and the other for the execute-stage: > > > > > scripts/hbase_upgrade.py > take_snapshot > > > > > This seems a bit confusing. It would also require changes to the actual > upgrade xml when now we would otherwise not require any changes to those > files. Using a tag like , does allow for some ambiguity in what the > actual order will be if several services specify that they should come after > the same step or service. I'm not convinced that it would really matter what > the order is. Yes, if a given service needs to be processed after another > then the order does matter but in that case the tag should be changed > to reflect that need. If we have two custom services that both need to be > started after HBASE then it shouldn't matter which one gets started before > the other. If there are dependencies between those two custom services then > one should state it should be started after the other. > > Nate Cole wrote: > You could use order only at the group level. I wouldn't expect that you > would be "injecting" execute-stage into an existing / > as those are already-vetted actions. There's no reason that two groups in a > row can't work against the same service/component. You're right that most > cases the order doesn't matter, but the design should allow for it. Hi Nate, There are several areas were order maybe needed: 1) ordering check elements within the prerequisite-checks tag 2) ordering group elements within the order tag 3) ordering execute-stage elements within a particular group tag (such as within prepare backups) 4) ordering service elements within a particular group tag (such as within the core slaves section) 5) ordering service elements within a priority tag (for service checks) I can see using an order attribute within the group elements but adding an order attribute to
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On March 2, 2016, 6:58 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/stacks/default_stack_advisor.py, line 35 > > <https://reviews.apache.org/r/44210/diff/1/?file=1275579#file1275579line35> > > > > Please include Srimanth Gunturi in the code review, thanks! > > Jayush Luniya wrote: > +1 on adding Srimanth to the CR. Could you please recommend who could be added for HAWQ and PXF to review those specific changes? - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review121687 ------- On April 12, 2016, 5:33 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 12, 2016, 5:33 p.m.) > > > Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-agent/pom.xml c2c993f > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py > 53591cd > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > f6f8cde > > ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HD
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On March 2, 2016, 9:37 p.m., jun aoki wrote: > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py, > > line 32 > > <https://reviews.apache.org/r/44210/diff/1/?file=1275516#file1275516line32> > > > > Could this inheritate a brand new DefaultServiceAdvisor instead of the > > stack advisor? service and stack are different hierarchy and I feel somehow > > wrong if a service inheritate a stack. > > Tim Thorpe wrote: > Point well taken. I'll look at refactoring it that way but really would > prefer to get the code in the current way first. There are code changes to > the stack advisor files on a daily basis and the longer it waits the more > merges will be needed. I will already be refactoring the code to split the > test scripts to be on the service level rather than the stack level. I will > attempt to do both of these tasks at the same time. > > Matt wrote: > May be this should inherit from service_advisor for that service under > common-services? > > Tim Thorpe wrote: > Hi Matt, so you are recommending that the > resources/stacks/service_advisor.py be moved to > resources/common-services/service_advisor.py? Probably a more logical place to put it would be under resources/scripts - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review121720 --- On April 12, 2016, 5:33 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 12, 2016, 5:33 p.m.) > > > Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-agent/pom.xml c2c993f > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py > 53591cd > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py > PRE-CREATION > > ambari-server/src/mai
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 5, 2016, 5:32 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py, > > line 1 > > <https://reviews.apache.org/r/44210/diff/1/?file=1275506#file1275506line1> > > > > Naming each service_advisor script named with > > stackname-version-servicename doesnt quite feel elegant. Simply > > service_advisor.py is what should be there at service-level. > > > > Also, we should do cross-stack references. When we bundle a stack in an > > Ambari release, we only bundle one-stack. So for BIGTOP, the HDP stack > > directory will not be present. Otherwise if you include both HDP and BIGTOP > > in the Ambari release, in the UI, you will see both BIGTOP and HDP stack > > options. I have figured out a solution for this. So the naming of the service advisor file would always be service_advisor.py without being prefaced with the stack name, version and service name. Since the stack advisors are no longer being refactored, we will only allow for service advisors to be plugged in, there will be no dependence of BIGTOP to HDP. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review127148 ------- On March 1, 2016, 4:53 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated March 1, 2016, 4:53 p.m.) > > > Review request for Ambari, Sumit Mohanty and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-agent/pom.xml c2c993f > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py > 53591cd > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On March 2, 2016, 4:56 a.m., Matt wrote: > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py, > > line 1 > > <https://reviews.apache.org/r/44210/diff/1/?file=1275503#file1275503line1> > > > > If I were to add HAWQ 2.0.0's metainfo.xml under HDP 2.3 stack, and say > > that it extends from common-services, will this service_advisor be honored? > > Tim Thorpe wrote: > I had some difficulty with HAWQ and PXF because they are not actually in > the stack and can be added by installing a separate RPM. In order for HAWQ > and PXF to inherit the service_advisor.py files from common-services, they > would need to implement mostly empty service advisors under the > //services/ folder that explicitly inherit from the > ones in common-services. > > Matt wrote: > Is there a way we can avoid having this empty service_advisor? > > service X in common-services has a service_advisor > service X in HDP 2.2 has no service_advisor - should inherit from > common-services > service X in HDP 2.3 has service_advisor - should inherit from > common-services + add new recommendations/validations from HDP 2.3 X's > service_advisor > > Something similar to have themes can be managed or how metainfo.xml has > extends property for stack definiton Hi Matt, I can add the service advisor in the code that reads the stack/services from the file system. This would allow me to know which service_advisor.py file should be used (either because it is inherited or because it is defined in the current stack version). This information would then need to be passed to the stack advisor in the services.json. I don't think the service advisor should handle implicite inheritance between stack versions by that I mean in that your example: service X in common-services has a service_advisor service X in HDP 2.2 has no service_advisor - should inherit from common-services service X in HDP 2.3 has service_advisor - only inherits from common-services if the service_advisor.py adds that logic Basically this means that service X in HDP 2.3 would have 2 options to replace the service_advisor.py in common-services or inherit from it and add new recommendations/validations. > On March 2, 2016, 4:56 a.m., Matt wrote: > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py, > > line 1 > > <https://reviews.apache.org/r/44210/diff/1/?file=1275515#file1275515line1> > > > > If the service_advisor is under HDP 2.0.6's GANGLIA service folder, it > > is not required to repeat the stack name, the version and the service on > > the filename, for every combination of stack, version and service. > > > > This would be good enough: > > HDP/2.0.6/services/GANGLIA/service_advisor.py > > Tim Thorpe wrote: > I tried having the name as you stated but had a problem with python. For > example in 2.3/services/HDFS, I have the following: > > SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) > PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/HDFS/') > sys.path.append(PARENT_DIR) > from service_advisor_HDP22HDFS import * > > If the file name is the same in 2.3 and 2.2, then "from service_advisor > import *" could reference the 2.2 or the 2.3 file. I tried it with import > HDP22HDFSServiceAdvisor as well and both of them failed to find the correct > parent class. > > The other way I could have solved this is to load all the service advisor > files from the default_stack_advisor.py walking my way up the stack. This > would have allowed you to avoid the inheritance code in the > service_advisor.py files but it would restrict you from inheriting from > outside of your stack (unless at that point you started including the > inheritance code and the change of file names). I have figured out a solution for this. So the naming of the service advisor file would always be service_advisor.py without being prefaced with the stack name, version and service name. > On March 2, 2016, 4:56 a.m., Matt wrote: > > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py, line > > 22 > > <https://reviews.apache.org/r/44210/diff/1/?file=1275586#file1275586line22> > > > > If all the stack_advisor.py files are being refactored into service > > level advisors, the unit tests should be at service level as well. > > > > This would bring better modularity for introducing a new service: > > - Add a new service to the stack > > - Implement the service advisor for that service > > - Implement unit test for the se
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
(and the need to make changes in multiple places) 2) Inherit the custom service advisor classes from the default stack advisor and live with the fact that a service is inheriting from the stack. (See one of the comments above by Jun Aoki) - Tim ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review127197 --- On March 1, 2016, 4:53 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated March 1, 2016, 4:53 p.m.) > > > Review request for Ambari, Sumit Mohanty and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-agent/pom.xml c2c993f > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py > 53591cd > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > f6f8cde > > ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.1/servic
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
/test_stack_advisor.py 8932bde ambari-server/src/test/resources/stacks/XYZ/1.0.0/services/YARN/service_advisor_XYZ100YARN.py PRE-CREATION ambari-server/src/test/resources/stacks/XYZ/1.0.0/services/stack_advisor.py ba140bb ambari-server/src/test/resources/stacks/XYZ/1.0.1/services/YARN/service_advisor_XYZ101YARN.py PRE-CREATION ambari-server/src/test/resources/stacks/XYZ/1.0.1/services/stack_advisor.py 74a4b31 ambari-server/src/test/resources/stacks/old/HDP206stack_advisor.py PRE-CREATION ambari-server/src/test/resources/stacks/old/HDP21stack_advisor.py PRE-CREATION ambari-server/src/test/resources/stacks/old/HDP22stack_advisor.py PRE-CREATION ambari-server/src/test/resources/stacks/old/HDP23stack_advisor.py PRE-CREATION ambari-server/src/test/resources/stacks/old/HDP24stack_advisor.py PRE-CREATION ambari-server/src/test/resources/stacks/old/HDP25stack_advisor.py PRE-CREATION ambari-server/src/test/resources/stacks/old/HDP26stack_advisor.py PRE-CREATION ambari-server/src/test/resources/stacks/old/stack_advisor.py PRE-CREATION Diff: https://reviews.apache.org/r/44210/diff/ Testing (updated) --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On March 2, 2016, 9:37 p.m., jun aoki wrote: > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py, > > line 32 > > <https://reviews.apache.org/r/44210/diff/1/?file=1275516#file1275516line32> > > > > Could this inheritate a brand new DefaultServiceAdvisor instead of the > > stack advisor? service and stack are different hierarchy and I feel somehow > > wrong if a service inheritate a stack. > > Tim Thorpe wrote: > Point well taken. I'll look at refactoring it that way but really would > prefer to get the code in the current way first. There are code changes to > the stack advisor files on a daily basis and the longer it waits the more > merges will be needed. I will already be refactoring the code to split the > test scripts to be on the service level rather than the stack level. I will > attempt to do both of these tasks at the same time. > > Matt wrote: > May be this should inherit from service_advisor for that service under > common-services? Hi Matt, so you are recommending that the resources/stacks/service_advisor.py be moved to resources/common-services/service_advisor.py? - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review121720 ------- On April 12, 2016, 5:33 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 12, 2016, 5:33 p.m.) > > > Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-agent/pom.xml c2c993f > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py > 53591cd > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py > PRE-CREATION > > ambari-server/src
Re: Review Request 44687: Update RCO : PXF should start after HDFS
> On March 11, 2016, 9:06 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json,
> > line 14
> > <https://reviews.apache.org/r/44687/diff/1/?file=1295169#file1295169line14>
> >
> > RCO can be defined at service level
> > (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating
> > RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the
> > service?
>
> bhuvnesh chaudhary wrote:
> Thanks Jayush for the suggestion, let me try that.
> However, RCO at service level will be applicable only for trunk, for 2.2
> branch we still need to update HDP rco. correct ?
> If so i will remove trunk tag from the list of branches, and keep this
> pathc for 2.2.0 only.
>
> bhuvnesh chaudhary wrote:
> Tried putting role_command_order.json under
> /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json.
> However, it appears that its not loaded due to which HAWQMASTER was started
> before NAMENODE and it failed. Same for PXF.
> Here are the contents of RCO:
> [root@c6401 ~]# cat
> /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json
> {
> "_comment" : "Record format:",
> "_comment" : "blockedRole-blockedCommand:
> [blockerRole1-blockerCommand1, blockerRole2-blockerCommand2, ...]",
> "general_deps" : {
> "_comment" : "dependencies for all cases",
> "HAWQMASTER-START" : ["NAMENODE-START", "DATANODE-START",
> "NODEMANAGER-START"],
> "HAWQSTANDBY-START" : ["HAWQMASTER-START"],
> "HAWQSEGMENT-START" : ["HAWQMASTER-START", "HAWQSTANDBY-START"],
> "HAWQ_SERVICE_CHECK-SERVICE_CHECK" : ["HAWQSEGMENT-START",
> "HDFS_SERVICE_CHECK-SERVICE_CHECK", "YARN_SERVICE_CHECK-SERVICE_CHECK",
> "PXF_SERVICE_CHECK-SERVICE_CHECK"]
> }
> }
>
> Checked ambari-server.log, it does not show that this RCO is loaded but
> shows the others under /var/lib/ambari-server/resources/stacks which are
> loaded, will need to debug further for the root cause.
> Adding Tim Thorpe as well to the reviewers list.
>
> Jayush Luniya wrote:
> @Bhuvnesh
> Go ahead with your changes without the refactoring in trunk and 2.2
> branch. I dont want to block you on it. Lets handle the refactoring in a
> separate JIRA. @Tim Thorpe, can you check if anything is wrong with the
> service-level RCO?
>
> bhuvnesh chaudhary wrote:
> Thank you Jayush, will go ahead with the change.
When I added the RCO change for putting it at the service level, I only tested
with services defined in the stack not having inherited the RCO from
common-services. I'll look into it some more and see if I can figure out what
the problem might be. Sorry was on vacation for a week.
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
---
On March 11, 2016, 9:50 p.m., bhuvnesh chaudhary wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> ---
>
> (Updated March 11, 2016, 9:50 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya,
> Matt, Oleksandr Diachenko, and Tim Thorpe.
>
>
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Update RCO to ensure that PXF is started after HDFS.
> PXF connects to namenode during startup and retries couple of times until
> namenode has come up.
>
>
> Diffs
> -
>
> ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json
> b28f2a9
>
> Diff: https://reviews.apache.org/r/44687/diff/
>
>
> Testing
> ---
>
> yes. manually.
>
>
> Thanks,
>
> bhuvnesh chaudhary
>
>
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On March 22, 2016, 6:27 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/ModuleFileUnmarshaller.java, > > line 31 > > <https://reviews.apache.org/r/45169/diff/1/?file=1310659#file1310659line31> > > > > Please give me some time to go through the design review and code > > review. > > > > Thank you, > > Alejandro No rush. Thanks for looking into it. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review124850 ------- On March 22, 2016, 6:18 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated March 22, 2016, 6:18 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate > Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/ModuleFileUnmarshaller.java > 9e2f997 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > c552e41 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/config-upgrade.xml > 440bd50 > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/falcon.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/hbase.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/hdfs.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/hive.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/kafka.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/knox.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/oozie.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/ranger.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/storm.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/tez.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/yarn.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml > 430114b > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/falcon.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/flume.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/hbase.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/hdfs.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/hive.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/kafka.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgra
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
to the JIRA. Not sure how to create a review board with a design doc instead of a patch file. Thanks, Tim Thorpe
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On March 29, 2016, 12:49 p.m., Nate Cole wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java, > > lines 535-547 > > <https://reviews.apache.org/r/45169/diff/1/?file=1310660#file1310660line535> > > > > What exactly is the use case for marshalling back to the filesystem? > > Filesystem access is read-only when running Ambari as non-root, so this > > code will not work. If you are looking for merge-ordering or something to > > make an "uber upgrade pack", then that can be done at orchestration or keep > > it in-memory. Hi Nate, the purpose was really for testing. I marshalled the results file back to the FS so I could compare the results with the original upgrade xml. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review125862 ----------- On March 22, 2016, 6:40 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated March 22, 2016, 6:40 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate > Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/ModuleFileUnmarshaller.java > 9e2f997 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > c552e41 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/config-upgrade.xml > 440bd50 > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/falcon.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/hbase.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/hdfs.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/hive.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/kafka.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/knox.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/oozie.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/ranger.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/storm.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/tez.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/configs/yarn.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml > 430114b > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/falcon.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/flume.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/hbase.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2/hdfs.xml > PRE-CREATION > > ambari-s
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> >
>
> Alexander Denissov wrote:
> So, what will be the rules for SA logic inheritance ?
>
> When MYSERVICE/2.0.0 ships and needs to reuse the stack_advisor logic
> from MYSERVICE/1.0.0 -- will we:
> - rely on stack inheritance ?
> - rely on SA inheritance within MYSERVICE ?
> - combination of these ?
There will be no implicit inheritance. If you want to use inheritance you can
but it will be left up to the service to handle it with code like:
SCRIPT_DIR = os.path.dirname(os.path.abspath(file))
PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')
try:
with open(PARENT_FILE, 'rb') as fp:
service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE,
('.py', 'rb', imp.PY_SOURCE))
except Exception as e:
traceback.print_exc()
print "Failed to load parent"
class
HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
-----------
On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
>
> (Updated April 22, 2016, 6:27 p.m.)
>
>
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
>
>
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the stack advisor is defined under each stack version such as
> HDP/2.3. The problem with this is that it restricts the services that can be
> added to the stack. If a custom service is to be added, they would need to
> modify the stack advisor. If the configuration recommendation and validation
> can be done at the service level then the custom service could just include
> their own recommendations and validations separately.
>
>
> Diffs
> -
>
> ambari-server/src/main/assemblies/server.xml e1a4919
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
> df65010
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
> 00c8696
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
> ca1968e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
> 6c6fa91
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 636de37
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> d574d60
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> b7e09a9
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> d27e52a
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> db95fec
>
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
> PRE-CREATION
> ambari-server/src/main/resources/properties.json eac0dbd
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
> 0130483
> ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py
> 3a65541
> ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION
> ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25
> ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py
> 6c9fd46
>
> Diff: https://reviews.apache.org/r/44210/diff/
>
>
> Testing
> ---
>
> Ran all the non java unit tests.
>
> Total run:945
> Total errors:0
> Total failures:0
>
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure
> their service advisors were called.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> >
>
> Alexander Denissov wrote:
> So, what will be the rules for SA logic inheritance ?
>
> When MYSERVICE/2.0.0 ships and needs to reuse the stack_advisor logic
> from MYSERVICE/1.0.0 -- will we:
> - rely on stack inheritance ?
> - rely on SA inheritance within MYSERVICE ?
> - combination of these ?
>
> Tim Thorpe wrote:
> There will be no implicit inheritance. If you want to use inheritance
> you can but it will be left up to the service to handle it with code like:
>
> SCRIPT_DIR = os.path.dirname(os.path.abspath(file))
> PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
> PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')
>
> try:
> with open(PARENT_FILE, 'rb') as fp:
> service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE,
> ('.py', 'rb', imp.PY_SOURCE))
> except Exception as e:
> traceback.print_exc()
> print "Failed to load parent"
>
>
> class
> HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):
So I guess the answer to your question is none of those.
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---
On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
>
> (Updated April 22, 2016, 6:27 p.m.)
>
>
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
>
>
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the stack advisor is defined under each stack version such as
> HDP/2.3. The problem with this is that it restricts the services that can be
> added to the stack. If a custom service is to be added, they would need to
> modify the stack advisor. If the configuration recommendation and validation
> can be done at the service level then the custom service could just include
> their own recommendations and validations separately.
>
>
> Diffs
> -
>
> ambari-server/src/main/assemblies/server.xml e1a4919
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
> df65010
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
> 00c8696
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
> ca1968e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
> 6c6fa91
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 636de37
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> d574d60
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> b7e09a9
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> d27e52a
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> db95fec
>
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
> PRE-CREATION
> ambari-server/src/main/resources/properties.json eac0dbd
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
> 0130483
> ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py
> 3a65541
> ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION
> ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25
> ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py
> 6c9fd46
>
> Diff: https://reviews.apache.org/r/44210/diff/
>
>
> Testing
> ---
>
> Ran all the non java unit tests.
>
> Total run:945
> Total errors:0
> Total failures:0
>
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure
> their service advisors were called.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> > line 44
> > <https://reviews.apache.org/r/44210/diff/2/?file=1348344#file1348344line44>
> >
> > *return componentName in ('HAWQMASTER', 'HAWQSTANDBY')*
That works better, think I just copied and pasted it from a different method.
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130191
-------
On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
>
> (Updated April 22, 2016, 6:27 p.m.)
>
>
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
>
>
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the stack advisor is defined under each stack version such as
> HDP/2.3. The problem with this is that it restricts the services that can be
> added to the stack. If a custom service is to be added, they would need to
> modify the stack advisor. If the configuration recommendation and validation
> can be done at the service level then the custom service could just include
> their own recommendations and validations separately.
>
>
> Diffs
> -
>
> ambari-server/src/main/assemblies/server.xml e1a4919
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
> df65010
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
> 00c8696
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
> ca1968e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
> 6c6fa91
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 636de37
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> d574d60
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> b7e09a9
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> d27e52a
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> db95fec
>
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
> PRE-CREATION
> ambari-server/src/main/resources/properties.json eac0dbd
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
> 0130483
> ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py
> 3a65541
> ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION
> ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25
> ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py
> 6c9fd46
>
> Diff: https://reviews.apache.org/r/44210/diff/
>
>
> Testing
> ---
>
> Ran all the non java unit tests.
>
> Total run:945
> Total errors:0
> Total failures:0
>
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure
> their service advisors were called.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/ --- (Updated April 22, 2016, 4:39 p.m.) Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Bugs: AMBARI-15226 https://issues.apache.org/jira/browse/AMBARI-15226 Repository: ambari Description --- Currently the stack advisor is defined under each stack version such as HDP/2.3. The problem with this is that it restricts the services that can be added to the stack. If a custom service is to be added, they would need to modify the stack advisor. If the configuration recommendation and validation can be done at the service level then the custom service could just include their own recommendations and validations separately. Diffs (updated) - ambari-server/src/main/assemblies/server.xml e1a4919 ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java df65010 ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 00c8696 ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java ca1968e ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java 6c6fa91 ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 636de37 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java d574d60 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java b7e09a9 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java d27e52a ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java db95fec ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/properties.json eac0dbd ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 0130483 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 3a65541 ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 6c9fd46 Diff: https://reviews.apache.org/r/44210/diff/ Testing --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 5:33 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/stacks/stack_advisor.py, line 522 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348350#file1348350line522> > > > > loadServiceAdvisor() instead? If I follow the tradition of the scripts/stack_advisor.py, I guess I should call it def instantiateServiceAdvisor(). Any one of those names are fine by me. create or instantiate are more accurate names I think, because it actually does both load the script and create an instance. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review130131 --- On April 22, 2016, 4:39 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 22, 2016, 4:39 p.m.) > > > Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, > and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > d574d60 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > db95fec > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 0130483 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 3a65541 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 6c9fd46 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 5:18 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py, > > line 19 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348344#file1348344line19> > > > > Please add Pivotal folks to the review to look at this. Also since the > > patch is a bit old, please verify and revise the patch to include any > > latest changes. I will update the patch to the latest once I get all the feedback including the Pivotal folks. Otherwise, I'm going to be updating the patch many times before I get to the point where it can be accept for commit. Can you please let me know who I should add from Pivotal? Thanks - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review130126 ------- On April 22, 2016, 4:39 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 22, 2016, 4:39 p.m.) > > > Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, > and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > d574d60 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > db95fec > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 0130483 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 3a65541 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 6c9fd46 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 5:10 p.m., Srimanth Gunturi wrote: > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java, > > line 77 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348336#file1348336line77> > > > > Trying to think if we really need to expose the 'advisor_name' and > > 'advisor_path' in the stack-service API response? The files will be at > > known locations anyways and used when available... similar to the stack's > > advisor_path/advisor_name. > > > > I am thinking this can work without adding these 2 properties to a > > stack-service. The reason this works for the stack is because the stack_advisor.py in the scripts directory attempts to load all the stack_advisor.py files. It uses the stack name, stack version and parent versions to determine what to load. It starts with the oldest version and keeps loading until the most recent version. In order to do this for services, we'd need to know the some sort of information to determine what to load from where. I could have added something similar so that it would look into the stack versions and then somehow look to common services if that was required. In my code I was testing with HAWQ and PXF with their service advisor's being loaded from the common-services directory. It seemed much easier to me at least to get the service_advisor.py location and calculate the advisor name in the java code when reading the stack and services (StackManager/StackModule/StackDirectory etc...). This way I know exactly where to load the py file and exactly what class name to use. > On April 22, 2016, 5:10 p.m., Srimanth Gunturi wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java, > > line 88 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348338#file1348338line88> > > > > I think we can make this work without adding new properties to > > stack-service resource. see my comment above. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review130123 --- On April 22, 2016, 4:39 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 22, 2016, 4:39 p.m.) > > > Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, > and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > d574d60 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > db95fec > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 5:15 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/stacks/service_advisor.py, line 24 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348349#file1348349line24> > > > > Add documentation for all functions here. I will make sure I document all the functions in the service advisor. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review130124 --- On April 22, 2016, 4:39 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 22, 2016, 4:39 p.m.) > > > Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, > and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > d574d60 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > db95fec > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 0130483 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 3a65541 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 6c9fd46 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 5:58 p.m., Jayush Luniya wrote: > > ambari-server/src/main/resources/stacks/stack_advisor.py, line 473 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348350#file1348350line473> > > > > I might be missing something, but it doesnt look like we are using > > getHostsForSlaveComponent Yep I missed something in that patch, when I updated an older patch over some later code. The createComponentLayoutRecommendations code should have the client/slave section like this: for service in services["services"]: slaveClientComponents = [component for component in service["components"] if self.isSlaveComponent(component) or self.isClientComponent(component)] serviceAdvisor = self.createServiceAdvisor(service) for component in slaveClientComponents: componentName = component["StackServiceComponents"]["component_name"] hostsForComponent = [] if serviceAdvisor is None: hostsForComponent = self.getHostsForSlaveComponent(services, hosts, component, hostsList, hostsComponentsMap) else: hostsForComponent = serviceAdvisor.getHostsForSlaveComponent(self, services, hosts, component, hostsList, hostsComponentsMap) - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review130138 ------- On April 22, 2016, 4:39 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 22, 2016, 4:39 p.m.) > > > Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, > and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > d574d60 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > db95fec > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 0130483 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 3a65541 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 6c9fd46 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py,
> > line 31
> > <https://reviews.apache.org/r/44210/diff/2/?file=1348345#file1348345line31>
> >
> > Lets say we have
> > common-services/MYSERVICE/1.0.0
> > |__ service_advisor.py
> > common-services/MYSERVICE/2.0.0 (<- extends
> > common-services/MYSERVICE/1.0.0)
> > |__ service_advisor.py
> >
> > stacks/HDP/2.3/services/MYSERVICE (<- extends
> > common-services/MYSERVICE/1.0.0)
> > |__ service_advisor.py
> >
> > stacks/HDP/2.5/services/MYSERVICE (<- extends
> > common-services/MYSERVICE/2.0.0)
> > |__ service_advisor.py
> >
> > I believe we wont support any sort of inheritance between them? I am ok
> > with keeping things simple right now but we should call out this limitation
> > and that there could be certain duplication.
>
> Jayush Luniya wrote:
> Sorry the formatting went off after publishing :(
Hi Jayush, you are right there won't be any automatic inheritance but I don't
think that's really a bad thing. You can still have explicit inheritance with
code like this:
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')
try:
with open(PARENT_FILE, 'rb') as fp:
service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE,
('.py', 'rb', imp.PY_SOURCE))
except Exception as e:
traceback.print_exc()
print "Failed to load parent"
class
HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---
On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
>
> (Updated April 22, 2016, 4:39 p.m.)
>
>
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi,
> and Yusaku Sako.
>
>
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the stack advisor is defined under each stack version such as
> HDP/2.3. The problem with this is that it restricts the services that can be
> added to the stack. If a custom service is to be added, they would need to
> modify the stack advisor. If the configuration recommendation and validation
> can be done at the service level then the custom service could just include
> their own recommendations and validations separately.
>
>
> Diffs
> -
>
> ambari-server/src/main/assemblies/server.xml e1a4919
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
> df65010
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
> 00c8696
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
> ca1968e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
> 6c6fa91
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 636de37
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> d574d60
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> b7e09a9
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> d27e52a
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> db95fec
>
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
> PRE-CREATION
> ambari-server/src/main/resources/properties.json eac0dbd
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
> 0130483
> ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py
> 3a65541
> ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION
> ambari-server/src/main/resources/stacks/stack_adviso
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> > line 118
> > <https://reviews.apache.org/r/44210/diff/2/?file=1348344#file1348344line118>
> >
> > There is some additional logic for HAWQ mentioned in
> > recommendHDFSConfigurations:
> > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py#L368-#L370
> >
> >
> > I believe this would still work from the HDP 2.3 stack advisor. Would
> > you like to move it here?
>
> Tim Thorpe wrote:
> I was a bit hesitant to make changes to the functions for the other
> components but it definitely could be.
So I made the change and ran the stack manager python tests and all of them
passed except one:
FAIL: test_recommendHAWQConfigurations
(test_stack_advisor.TestHDP23StackAdvisor)
--
Traceback (most recent call last):
File
"/home/tim/dev/workspaces/15226-test/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py",
line 2047, in test_recommendHAWQConfigurations
'hawq-site': {'properties': {}}})
AssertionError: {'hdfs-client': {'properties':
{'output.replace-datanode-on-failure': 'false'}}, 'hawq-site': {'properties':
{}}, 'hdfs-site': {'properties': {'dfs.allow.truncate': 'true'}}} !=
{'hdfs-client': {'properties': {'output.replace-datanode-on-failure':
'false'}}, 'hawq-site': {'properties': {}}}
I could modify the test to include hdfs-site's dfs.allow.truncate = true.
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130191
---
On April 27, 2016, 8:48 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
>
> (Updated April 27, 2016, 8:48 p.m.)
>
>
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
>
>
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the stack advisor is defined under each stack version such as
> HDP/2.3. The problem with this is that it restricts the services that can be
> added to the stack. If a custom service is to be added, they would need to
> modify the stack advisor. If the configuration recommendation and validation
> can be done at the service level then the custom service could just include
> their own recommendations and validations separately.
>
>
> Diffs
> -
>
> ambari-server/src/main/assemblies/server.xml e1a4919
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
> df65010
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
> 00c8696
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
> ca1968e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
> 6c6fa91
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 636de37
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> 356adb1
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> b7e09a9
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> d27e52a
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> 5a2bf84
>
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
> PRE-CREATION
> ambari-server/src/main/resources/properties.json eac0dbd
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
> 6b3b1f5
> ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py
> 9f77129
> ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION
> ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e
> ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py
> 2080c52
>
> Diff: https://reviews.apache.org/r/44210/diff/
>
>
> Testing
> ---
>
> Ran all the non java unit tests.
>
> Total run:945
> Total errors:0
> Total failures:0
>
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure
> their service advisors were called.
>
>
> Thanks,
>
> Tim Thorpe
>
>
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java, > > line 56 > > <https://reviews.apache.org/r/44210/diff/5/?file=1364835#file1364835line56> > > > > should we not have a convention at all and rely on property specified > > in metainfo.xml ? > > > > The service advisors can be called HAWQ200ServiceAdvisor for the > > service in common services and HDP23HAWQ200ServiceAdvisor for the version > > specific to HDP2.3 stack, if any. > > > > This is the argument about inheritance between advisors of different > > versions of service and stack. I debated about this but I don't see what you really gain by allowing the advisor name or file name to be specified in the metainfo.xml. Although the code there should be changed to serviceName + serviceVersion + "ServiceAdvisor". This would support the situation you mentioned in your email about adding HAWQ 2.1.0 to common-services and it inheriting from HAWQ200ServiceAdvisor. The file name should always be "service_advisor.py" much like we always have "stack_advisor.py". The class name of the service advisor depends on whether it is from common-services or from the stack. It should be ServiceAdvisor in common-services and ServiceAdvisor in the stack. > On April 29, 2016, 12:11 a.m., Alexander Denissov wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java, > > line 65 > > <https://reviews.apache.org/r/44210/diff/5/?file=1364838#file1364838line65> > > > > In case the service comes from the common services (like HAWQ) which > > can be versioned independently from the stack, I think we should include > > the service version: > > > > HDP23HAWQ200ServiceAdvisor when HDP2.3 uses HAWQ 2.0.0 > > > > HDP23HAWQ210ServiceAdvisor when HDP2.3 uses HAWQ 2.1.0 > > > > This example assumes that HAWQ 2.1.0 can be release before the next > > version of Ambari or stack can be released. I'm not really sure how this would help. Are you thinking that you'd want 2 service advisor classes defined in HDP/2.3/services/HAWQ/service_advisor.py or 2 service advisor files in HDP/2.3/services/HAWQ? I don't really understand why you'd want to do either of those. You are either using 200 or 210. I don't think it really matters if that switch is defined at the service_advisor.py level or the metainfo.xml level. So you'd define it as HDP23HAWQServiceAdvisor(service_advisor.HAWQ200ServiceAdvisor) or HDP23HAWQServiceAdvisor(service_advisor.HAWQ210ServiceAdvisor) > On April 29, 2016, 12:11 a.m., Alexander Denissov wrote: > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py, > > line 33 > > <https://reviews.apache.org/r/44210/diff/5/?file=1364840#file1364840line33> > > > > Should the name include the version, so the future versions can extend > > this: > > > > HAWQ200ServiceAdvisor ? > > > > Also applies to the file name, should we have > > HAWQ200_service_advisor.py ? I agree for common-services the service version should be included. I don't think there is a reason to have it in the file name. I had a previous comment that recommended the file name always remain the same. This is identical to the way stack_advisor.py is handled. > On April 29, 2016, 12:11 a.m., Alexander Denissov wrote: > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py, > > line 46 > > <https://reviews.apache.org/r/44210/diff/5/?file=1364840#file1364840line46> > > > > we have a case (not coded yet) that might require more flexible lgoic > > here for which we will need more metadata. When we do thin, we might extend > > the interface to take a list of current component layout. I think I would need to see a bit more about what you are thinking. I'm not sure that function would necessarily be the best place to have that logic. Either way, if in the future you need to change things, all you need to do is modify the stacks/stack_advisor.py to pass in the extra parameter(s), modify the stacks/service_advisor.py to accept the extra parameter(s) and then modify your service_advisor.py file(s) as well. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review131009 --- On April 28, 2016, 4:36 p.m., Tim Thorpe wrote: > > --- > This is an au
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 28, 2016, 10:54 p.m., Alexander Denissov wrote: > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py, > > line 110 > > <https://reviews.apache.org/r/44210/diff/3-5/?file=1363971#file1363971line110> > > > > if we make isLocalHost utility function available in another utility, > > then there will be no need to pass stackAdvisor as parameter to all the > > functions in a ServiceAdvisor, as it promotes "spagetti-code", since > > StackAdvisor calls into ServiceAdvisor and having the call coming in the > > opposite directions is not desired. I have been asked to keep this patch as minimal as possible to avoid destabilizing the code. I could have refactored many of the functions into a common file which both the stacks/stack_advisor.py and stacks/service_advisor.py could use but this would result in many more possible issues. To play it safe I passed in the stackAdvisor. In a previous iteration, when I had refactored most of the stack advisor even for the stack services like YARN, HDFS and HBASE, I found that sometimes I needed information in say the YARN service_advisor.py that was only available in the HDFS service_advisor.py. In that case, I would create an HDFS service advisor and call the required functions. Now with all of those functions being in the stackAdvisor, you can't do those sorts of things. This means that for some service, you might need a reference to the stackAdvisor. Obviously it needs to be used sparingly. In the future I hope to come back to this code and give a better separation between stack and service advisors. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review131002 ----------- On April 28, 2016, 4:36 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 28, 2016, 4:36 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 356adb1 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 5a2bf84 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 1680f21 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > d0ce196 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 2080c52 > > Diff:
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 11:19 p.m., Matt wrote: > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py, > > line 118 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348344#file1348344line118> > > > > There is some additional logic for HAWQ mentioned in > > recommendHDFSConfigurations: > > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py#L368-#L370 > > > > > > I believe this would still work from the HDP 2.3 stack advisor. Would > > you like to move it here? I was a bit hesitant to make changes to the functions for the other components but it definitely could be. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review130191 ------- On April 22, 2016, 6:27 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 22, 2016, 6:27 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > d574d60 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > db95fec > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 0130483 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 3a65541 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 6c9fd46 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/ --- (Updated April 29, 2016, 8:11 p.m.) Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Changes --- Did another quick change to change the common-services naming convention to include the service version number and remove lines 115 and 116 in the HAWQ service advisor: servicesList = [service["StackServices"]["service_name"] for service in services["services"]] if "HAWQ" in servicesList: Bugs: AMBARI-15226 https://issues.apache.org/jira/browse/AMBARI-15226 Repository: ambari Description --- Currently the stack advisor is defined under each stack version such as HDP/2.3. The problem with this is that it restricts the services that can be added to the stack. If a custom service is to be added, they would need to modify the stack advisor. If the configuration recommendation and validation can be done at the service level then the custom service could just include their own recommendations and validations separately. Diffs (updated) - ambari-server/src/main/assemblies/server.xml e1a4919 ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java df65010 ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 00c8696 ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java ca1968e ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java 6c6fa91 ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 636de37 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 356adb1 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java b7e09a9 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java d27e52a ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 5a2bf84 ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/properties.json eac0dbd ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 1680f21 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py f475798 ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 Diff: https://reviews.apache.org/r/44210/diff/ Testing --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java, > > line 56 > > <https://reviews.apache.org/r/44210/diff/5/?file=1364835#file1364835line56> > > > > should we not have a convention at all and rely on property specified > > in metainfo.xml ? > > > > The service advisors can be called HAWQ200ServiceAdvisor for the > > service in common services and HDP23HAWQ200ServiceAdvisor for the version > > specific to HDP2.3 stack, if any. > > > > This is the argument about inheritance between advisors of different > > versions of service and stack. > > Tim Thorpe wrote: > I debated about this but I don't see what you really gain by allowing the > advisor name or file name to be specified in the metainfo.xml. Although the > code there should be changed to serviceName + serviceVersion + > "ServiceAdvisor". This would support the situation you mentioned in your > email about adding HAWQ 2.1.0 to common-services and it inheriting from > HAWQ200ServiceAdvisor. > > The file name should always be "service_advisor.py" much like we always > have "stack_advisor.py". The class name of the service advisor depends on > whether it is from common-services or from the stack. It should be > ServiceAdvisor in common-services and > ServiceAdvisor in the stack. > > Tim Thorpe wrote: > When I scanned over the stacks/service_advisor.py, I noticed that the > naming convention was not documented. I added that in. > HAWQ200ServiceAdvisor for the service in common services > HDP23HAWQServiceAdvisor for the version specific to the HDP 2.3 stack > > Alexander Denissov wrote: > Can you please give an example of how the advisor path / name will be > specified in metainfo.xml ? > > Your naming convention makes sense. One case we were considering was when > we would support multiple versions of HAWQ for the same stack. That might > require changes to the service advisor. If the file is always called > service_advisor.py and the class is HDP23HAWQServiceAdvisor, then there might > be 2 versions of the file with the same name and the same class inside. The > file will need to be managed by stand-alone plugin (extension in later > design) along with metainfo.xml that goes in stacks/HDP/2.X/services/HAWQ The > difference probably would be in inheritance: > > HDP23HAWQServiceAdvisor (HAWQ200ServiceAdvisor) --> for the original > HDP23HAWQServiceAdvisor (HAWQ210ServiceAdvisor) --> for updated service > version Currently there is nothing in the metainfo.xml to specify the advisor name & path. The ServiceDirectory java code will detect the presence of service_advisor.py files in the root of the service folder (whether common-services or in the stack). It will use the existing inheritance method such that if the stack service doesn't contain the service_advisor.py file, it will inherit that file from common-services or its parent stack version (if the service exists there). The name will be calculated in that same java code based on the naming convention mentioned above. All inheritance from python file to python (as previously mentioned) will be handled by explicit inheritance included by the service developers. - Tim ----------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review131009 --- On April 28, 2016, 4:36 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 28, 2016, 4:36 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Di
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/ --- (Updated April 28, 2016, 4:34 p.m.) Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Bugs: AMBARI-15226 https://issues.apache.org/jira/browse/AMBARI-15226 Repository: ambari Description --- Currently the stack advisor is defined under each stack version such as HDP/2.3. The problem with this is that it restricts the services that can be added to the stack. If a custom service is to be added, they would need to modify the stack advisor. If the configuration recommendation and validation can be done at the service level then the custom service could just include their own recommendations and validations separately. Diffs (updated) - ambari-server/src/main/assemblies/server.xml e1a4919 ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java df65010 ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 00c8696 ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java ca1968e ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java 6c6fa91 ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 636de37 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 356adb1 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java b7e09a9 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java d27e52a ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 5a2bf84 ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/properties.json eac0dbd ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 1680f21 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py d0ce196 ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 Diff: https://reviews.apache.org/r/44210/diff/ Testing --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 22, 2016, 5:10 p.m., Srimanth Gunturi wrote: > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java, > > line 77 > > <https://reviews.apache.org/r/44210/diff/2/?file=1348336#file1348336line77> > > > > Trying to think if we really need to expose the 'advisor_name' and > > 'advisor_path' in the stack-service API response? The files will be at > > known locations anyways and used when available... similar to the stack's > > advisor_path/advisor_name. > > > > I am thinking this can work without adding these 2 properties to a > > stack-service. > > Tim Thorpe wrote: > The reason this works for the stack is because the stack_advisor.py in > the scripts directory attempts to load all the stack_advisor.py files. It > uses the stack name, stack version and parent versions to determine what to > load. It starts with the oldest version and keeps loading until the most > recent version. In order to do this for services, we'd need to know the some > sort of information to determine what to load from where. I could have added > something similar so that it would look into the stack versions and then > somehow look to common services if that was required. In my code I was > testing with HAWQ and PXF with their service advisor's being loaded from the > common-services directory. It seemed much easier to me at least to get the > service_advisor.py location and calculate the advisor name in the java code > when reading the stack and services (StackManager/StackModule/StackDirectory > etc...). This way I know exactly where to load the py file and exactly what > class name t o use. > > Srimanth Gunturi wrote: > Yes, I agree that it makes it easy to determine location of files. But if > you look at it from pure API perspective, it is not useful for any caller > except ambari-server - None of the callers have access to those paths/files > except ambari-server. It is ambari-server's internals exposed outside just > for its own consumption. Also, like the stack's service-advisor.py, the > service's files can be dynamically located and loaded. > > If it is a question of efficiency, we can cache the result so that for a > stack-version's service we only determine this once. > > My vote would be to not have these properties on the stack service's > response. Instead of adding them to the stack service's response, I can inject it only for the stack advisor calls. I'll work on making the change. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review130123 --- On April 22, 2016, 6:27 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 22, 2016, 6:27 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > d574d60 > > ambari-server/src/main/
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
the full upgrade xml to disk and then compare the results to the original upgrade xml. This review is mostly for the design doc which is attached to the JIRA. Not sure how to create a review board with a design doc instead of a patch file. Thanks, Tim Thorpe
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 5:39 p.m., Jayush Luniya wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java, > > line 844 > > <https://reviews.apache.org/r/45169/diff/2/?file=1382543#file1382543line844> > > > > The after tag is overloaded. Meaning it could mean insert this service > > after another service in some place and in some places it could mean insert > > this group after another group. I am wondering if there would be a case > > where we would need a combination (example: Create a new group that is not > > in the stack upgrade pack and add 2 new services to this group in a > > specific order). > > > > Instead of leaving the tag to interpretation, why not be > > explicit as , etc. That way we can > > support combinations > > Jayush Luniya wrote: > Example: > > > CORE_MASTERS > > MY_MASTER_1 > > > > > > CORE_MASTERS > MY_SERVICE_1 > > MY_MASTER_2 > > > > Alterative > > > CORE_MASTERS > > MY_MASTER_1 > > > > > > CORE_MASTERS > > MY_SERVICE_1 > MY_MASTER_2 > > > > Tim Thorpe wrote: > With custom services they should just be able to create two new groups > rather than try to combine them into one. If there really is a need for the > a new group that spans multiple services, then it is probably a candidate for > inclusion in the main upgrade xml. You can even include an empty group in > the main upgrade xml like: > > > > Jayush Luniya wrote: > RE: If there really is a need for the a new group that spans multiple > services, then it is probably a candidate for inclusion in the main upgrade > xml. > Well these services are add-on services meaning say not part of HDP stack > itself (example: HAWQ and PFX add-on services on HDP stack). So putting them > in stack upgrade pack wouldnt be right. > > Yes these can be in 2 separate groups, but that would be because of this > restriction. > > Besides its also confusing when we overload the same property element. > For example the group name in upgrade pack could be the service name itself > like below, so STORM could mean after service STORM or after > group STORM :) > > > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml#L356-L364 > > true > > NIMBUS > SUPERVISOR > STORM_UI_SERVER > DRPC_SERVER > > Currently, I'm looping through all the groups from the main upgrade xml. I add those to a <String, List> map. This means the group from the main upgrade xml (if there was one) will be the first group in the list. This is important because I know those groups are already internally ordered. By that I mean there services/priorities/stages do NOT need to be shifted in the overall order (all insertions into that order are determined by the subsequent groups in the list). Currently when I process the service upgade xml files, for each group if the group name already exists in the map then this means the group existed in the main upgrade xml and the entries (services/priorities/stages) will be inserted into the original entry order based on the tag. If the group name doesn't exist, it is added with its group as the first group in the list. In this case I never expect another group with the same name to be added to that list. The missing piece is with this change I need to reorder the List in the map to find a group that doesn't have an and if all groups do have that tag, then I guess I have an error situation. Adding explicit tags for and would make it easier to understand. I would prefer to limit it to two tags and rather than split the into separate tags for services/priorities/stages. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review133556 --- On May 18, 2016, 6:26 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 6:41 a.m., Jayush Luniya wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java, > > line 685 > > <https://reviews.apache.org/r/45169/diff/2/?file=1382543#file1382543line685> > > > > UGM? This was just for testing purposes. I'll remove the unnecessary logging. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review133501 ------- On May 16, 2016, 6:50 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated May 16, 2016, 6:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush > Luniya, and Nate Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 7f7a49e > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 8a7b42b > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > f781574 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > 13d5047 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 6129ec0 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 88f6e19 > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 43cefb9 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 6b74af0 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 9fb2bba > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 1cd2ffa > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > e3bc7a3 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 9c6a02d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > 1745de8 > > Diff: https://reviews.apache.org/r/45169/diff/ > > > Testing > --- > > Manual testing so far. I have the code read the upgrade xml and all of its > service specific xml files, built the upgrade pack and then write the full > upgrade xml to disk and then compare the results to the original upgrade xml. > > This review is mostly for the design doc which is attached to the JIRA. Not > sure how to create a review board with a design doc instead of a patch file. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 6:23 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml, > > line 166 > > <https://reviews.apache.org/r/45169/diff/2/?file=1382549#file1382549line166> > > > > Does the group name have to be unique now or is this only for the > > purposes of making debugging easier? In order to be able to add a custom component into the priority list of a particular service check step, there would need to be something distinct about them, either the name or the title. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review133587 ------- On May 16, 2016, 6:50 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated May 16, 2016, 6:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush > Luniya, and Nate Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 7f7a49e > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 8a7b42b > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > f781574 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > 13d5047 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 6129ec0 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 88f6e19 > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 43cefb9 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 6b74af0 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 9fb2bba > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 1cd2ffa > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > e3bc7a3 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 9c6a02d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > 1745de8 > > Diff: https://reviews.apache.org/r/45169/diff/ > > > Testing > --- > > Manual testing so far. I have the code read the upgrade xml and all of its > service specific xml files, built the upgrade pack and then write the full > upgrade xml to disk and then compare the results to the original upgrade xml. > > This review is mostly for the design doc which is attached to the JIRA. Not > sure how to create a review board with a design doc instead of a patch file. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 6:33 a.m., Jayush Luniya wrote: > > Can you add unit test coverage? > > Jayush Luniya wrote: > We should have unit tests in particular to validate incorrectly authored > service upgrade packs. What happens if we add a circular dependency (example: > KAFKA is marked with KNOX and KNOX is marked with > KAFKA) > > Jayush Luniya wrote: > Also we should document any unsupported use cases. Added unit test for both a possible case and for the circular dependency. Really the circular dependency case also covers when you have FOO after BAR but BAR is not included. Where should the documentation go? - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review133499 ------- On May 16, 2016, 6:50 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated May 16, 2016, 6:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush > Luniya, and Nate Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 7f7a49e > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 8a7b42b > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > f781574 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > 13d5047 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 6129ec0 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 88f6e19 > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 43cefb9 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 6b74af0 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 9fb2bba > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 1cd2ffa > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > e3bc7a3 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 9c6a02d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > 1745de8 > > Diff: https://reviews.apache.org/r/45169/diff/ > > > Testing > --- > > Manual testing so far. I have the code read the upgrade xml and all of its > service specific xml files, built the upgrade pack and then write the full > upgrade xml to disk and then compare the results to the original upgrade xml. > > This review is mostly for the design doc which is attached to the JIRA. Not > sure how to create a review board with a design doc instead of a patch file. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 6:23 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java, > > line 485 > > <https://reviews.apache.org/r/45169/diff/2/?file=1382542#file1382542line485> > > > > May want to check that upgradeFile is not null before calling > > getAbsolutePath() on it, or return null if upgradeFile is null at the start. This should never be null because of how the method is called. A list of files in the upgrade folder is processed and the one that matches config-upgrade.xml is parsed. But I added the extra check just in case. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review133587 ------- On May 16, 2016, 6:50 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated May 16, 2016, 6:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush > Luniya, and Nate Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 7f7a49e > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 8a7b42b > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > f781574 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > 13d5047 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 6129ec0 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 88f6e19 > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 43cefb9 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 6b74af0 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 9fb2bba > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 1cd2ffa > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > e3bc7a3 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 9c6a02d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > 1745de8 > > Diff: https://reviews.apache.org/r/45169/diff/ > > > Testing > --- > > Manual testing so far. I have the code read the upgrade xml and all of its > service specific xml files, built the upgrade pack and then write the full > upgrade xml to disk and then compare the results to the original upgrade xml. > > This review is mostly for the design doc which is attached to the JIRA. Not > sure how to create a review board with a design doc instead of a patch file. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 2:30 p.m., Nate Cole wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java, > > lines 822-831 > > <https://reviews.apache.org/r/45169/diff/2/?file=1382543#file1382543line822> > > > > Visitor or abstract method? Added the mergeRegularGroupingByName to the Grouping class as merge(Iterator) throws AmbariException The other methods were added to ClusterGrouping and ServiceCheckGrouping respectively, overridding the merge method from the Grouping class. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review133533 ------- On May 16, 2016, 6:50 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated May 16, 2016, 6:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush > Luniya, and Nate Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 7f7a49e > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 8a7b42b > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > f781574 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > 13d5047 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 6129ec0 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 88f6e19 > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 43cefb9 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 6b74af0 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 9fb2bba > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 1cd2ffa > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > e3bc7a3 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 9c6a02d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > 1745de8 > > Diff: https://reviews.apache.org/r/45169/diff/ > > > Testing > --- > > Manual testing so far. I have the code read the upgrade xml and all of its > service specific xml files, built the upgrade pack and then write the full > upgrade xml to disk and then compare the results to the original upgrade xml. > > This review is mostly for the design doc which is attached to the JIRA. Not > sure how to create a review board with a design doc instead of a patch file. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 5:39 p.m., Jayush Luniya wrote: > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java, > > line 829 > > <https://reviews.apache.org/r/45169/diff/2/?file=1382543#file1382543line829> > > > > What about RestartGrouping, ColocatedGrouping, StartGrouping, > > StopGrouping, UpdateStackGrouping? How about parent.merge(iterator) instead? Implemented parent.merge(iterator), RestartGrouping, ColocatedGrouping, StartGrouping, StopGrouping, UpdateStackGrouping just all use the default behavior like what was happening before with mergeRegularGrouping. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/#review133556 ------- On May 16, 2016, 6:50 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45169/ > --- > > (Updated May 16, 2016, 6:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush > Luniya, and Nate Cole. > > > Bugs: AMBARI-15388 > https://issues.apache.org/jira/browse/AMBARI-15388 > > > Repository: ambari > > > Description > --- > > Currently the upgrade is defined as a series of xml files specific to the > current stack version and the target stack version. Each upgrade xml defines > the overall sequence of the upgrade and what needs to be done for each > service. It would both easier to maintain and easier to add new services, if > the services themselves could specify what should be done during their > upgrade. > > There are two ways to make these changes, the alternate approach would be to > only make the java changes and not split the upgrade xml files. This would > still allow new services to add themselves into the upgrade. The benefit of > this is that for the stack services you only have one upgrade xml file. The > problem with that is it is easier for a particular service to have > unintentional changes between upgrade xml files. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 7f7a49e > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 8a7b42b > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > f781574 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > 13d5047 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 6129ec0 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 88f6e19 > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 43cefb9 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > b860731 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > 67d7fdb > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java > 5cda422 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 6b74af0 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 9fb2bba > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 1cd2ffa > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > e3bc7a3 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 9c6a02d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > 1745de8 > > Diff: https://reviews.apache.org/r/45169/diff/ > > > Testing > --- > > Manual testing so far. I have the code read the upgrade xml and all of its > service specific xml files, built the upgrade pack and then write the full > upgrade xml to disk and then compare the results to the original upgrade xml. > > This review is mostly for the design doc which is attached to the JIRA. Not > sure how to create a review board with a design doc instead of a patch file. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/ --- (Updated May 13, 2016, 1:46 p.m.) Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole. Changes --- Added javadoc for UpgradePack changes Bugs: AMBARI-15388 https://issues.apache.org/jira/browse/AMBARI-15388 Repository: ambari Description --- Currently the upgrade is defined as a series of xml files specific to the current stack version and the target stack version. Each upgrade xml defines the overall sequence of the upgrade and what needs to be done for each service. It would both easier to maintain and easier to add new services, if the services themselves could specify what should be done during their upgrade. There are two ways to make these changes, the alternate approach would be to only make the java changes and not split the upgrade xml files. This would still allow new services to add themselves into the upgrade. The benefit of this is that for the stack services you only have one upgrade xml file. The problem with that is it is easier for a particular service to have unintentional changes between upgrade xml files. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 7f7a49e ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 8a7b42b ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java f781574 ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java 13d5047 ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 6129ec0 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java 88f6e19 ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 43cefb9 ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java b860731 ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 67d7fdb ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java 5cda422 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 6b74af0 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 9fb2bba ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 1cd2ffa ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml e3bc7a3 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 9c6a02d ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml 1745de8 Diff: https://reviews.apache.org/r/45169/diff/ Testing --- Manual testing so far. I have the code read the upgrade xml and all of its service specific xml files, built the upgrade pack and then write the full upgrade xml to disk and then compare the results to the original upgrade xml. This review is mostly for the design doc which is attached to the JIRA. Not sure how to create a review board with a design doc instead of a patch file. Thanks, Tim Thorpe
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/upgrades/upgrade_test_15388.xml PRE-CREATION Diff: https://reviews.apache.org/r/45169/diff/ Testing --- Manual testing so far. I have the code read the upgrade xml and all of its service specific xml files, built the upgrade pack and then write the full upgrade xml to disk and then compare the results to the original upgrade xml. This review is mostly for the design doc which is attached to the JIRA. Not sure how to create a review board with a design doc instead of a patch file. Thanks, Tim Thorpe
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
> On May 17, 2016, 2:30 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java,
> > lines 890-892
> > <https://reviews.apache.org/r/45169/diff/2/?file=1382543#file1382543line890>
> >
> > Really need to throw here or just log it and continue; ?
>
> Tim Thorpe wrote:
> When the StackManager encounters errors while reading the stack, it
> almost always ends up throwing an exception and ambari-server fails to start.
> I'm ok with just logging the issue for this and other errors but then you
> could think that everything is fine with the service specific upgrade xml
> unless you dig through the log.
>
> Nate Cole wrote:
> That's true, but in this case you aren't bound by reading the stack per
> se, you're acting on the applicability of an XML element. This is happening
> in mergeServiceCheckGrouping, implying you're only considering
> ServiceCheckGroupings anyway.
>
> If we actually used XSD (I have a future work-item for that) then sure,
> that could cause failure of stack. But in this case, I think a simple no-op
> continue; is fine.
Jayush suggested that I needed to make sure when merging groups that all the
groups were of the same type. So this applies to regular Grouping,
ClusterGrouping, ServiceCheckGrouping, etc...
List list = allGroupMap.get(group.name);
if (list.size() > 0) {
Grouping first = list.get(0);
if (!first.getClass().equals(group.getClass())) {
throw new AmbariException("Expected class: " + first.getClass() +
" instead of " + group.getClass());
}
}
allGroupMap.get(group.name).add(group);
Basically with the above code, the logging would never occur when merging
ServiceCheckGroupings. Do you think the above code should just log as well or
is it more critical because it applies to groups other than
ServiceCheckGroupings?
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45169/#review133533
---
On May 18, 2016, 1:19 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45169/
> ---
>
> (Updated May 18, 2016, 1:19 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush
> Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-15388
> https://issues.apache.org/jira/browse/AMBARI-15388
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the upgrade is defined as a series of xml files specific to the
> current stack version and the target stack version. Each upgrade xml defines
> the overall sequence of the upgrade and what needs to be done for each
> service. It would both easier to maintain and easier to add new services, if
> the services themselves could specify what should be done during their
> upgrade.
>
> There are two ways to make these changes, the alternate approach would be to
> only make the java changes and not split the upgrade xml files. This would
> still allow new services to add themselves into the upgrade. The benefit of
> this is that for the stack services you only have one upgrade xml file. The
> problem with that is it is easier for a particular service to have
> unintentional changes between upgrade xml files.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 7f7a49e
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> 8a7b42b
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> f781574
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java
> 13d5047
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java
> 5a18b3f
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> 88f6e19
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> 43cefb9
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
> b860731
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
> 3325469
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
> 67d7fdb
>
> ambari-server/src/main/java/org/apache/
Re: Review Request 45169: AMBARI-15388 - Upgrade XML should be pushed down as much as possible to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45169/ --- (Updated May 18, 2016, 6:26 p.m.) Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole. Changes --- Fixing issue exposed by UpgradeResourceProviderHDP22Test. Switched the target version in src/test/resources/stacks/HDP/2.2.0/upgrades/upgrade_test_15388.xml to 2.4 Bugs: AMBARI-15388 https://issues.apache.org/jira/browse/AMBARI-15388 Repository: ambari Description --- Currently the upgrade is defined as a series of xml files specific to the current stack version and the target stack version. Each upgrade xml defines the overall sequence of the upgrade and what needs to be done for each service. It would both easier to maintain and easier to add new services, if the services themselves could specify what should be done during their upgrade. There are two ways to make these changes, the alternate approach would be to only make the java changes and not split the upgrade xml files. This would still allow new services to add themselves into the upgrade. The benefit of this is that for the stack services you only have one upgrade xml file. The problem with that is it is easier for a particular service to have unintentional changes between upgrade xml files. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 7f7a49e ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 8a7b42b ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java f781574 ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java 13d5047 ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 5a18b3f ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java 88f6e19 ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 43cefb9 ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java b860731 ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java 3325469 ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java 67d7fdb ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java 5cda422 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 6b74af0 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 9fb2bba ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 1e040e6 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml e3bc7a3 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 6e27da6 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml d755516 ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerMiscTest.java dda1e7a ambari-server/src/test/java/org/apache/ambari/server/state/stack/UpgradePackTest.java 15be8b4 ambari-server/src/test/resources/stacks/HDP/2.0.5/services/HDFS/upgrades/HDP/2.2.0/upgrade_test_15388.xml PRE-CREATION ambari-server/src/test/resources/stacks/HDP/2.2.0/upgrades/upgrade_test_15388.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/repos/hdp.json PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/repos/repoinfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/repos/version-2.2.0.4-123.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/role_command_order.json PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/services/HDFS/metainfo.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/services/HDFS/upgrades/HDP/2.2.0/upgrade_test_15388.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/upgrades/config-upgrade.xml PRE-CREATION ambari-server/src/test/resources/stacks_with_upgrade_cycle/HDP/2.2.0/upgrades/upgrade_test_15388.xml PRE-CREATION Diff: https://reviews.apache.org/r/45169/diff/ Testing --- Manual testing so far. I have the code read the upgrade xml and all of its service specific xml files, built the upgrade pack and then write the full upgrade xml to disk and then compare the results to the original upgrade xml. This review is mostly for the design doc which is attached to the JIRA. Not sure how to create a review board with a design doc instead of a patch file. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/ --- (Updated May 3, 2016, 7:04 p.m.) Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Changes --- This fixes the issue Lav brought up about configuration validation not calling the service advisors. Bugs: AMBARI-15226 https://issues.apache.org/jira/browse/AMBARI-15226 Repository: ambari Description --- Currently the stack advisor is defined under each stack version such as HDP/2.3. The problem with this is that it restricts the services that can be added to the stack. If a custom service is to be added, they would need to modify the stack advisor. If the configuration recommendation and validation can be done at the service level then the custom service could just include their own recommendations and validations separately. Diffs (updated) - ambari-server/src/main/assemblies/server.xml e1a4919 ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java df65010 ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 00c8696 ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java ca1968e ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java 6c6fa91 ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 636de37 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 356adb1 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java b7e09a9 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java d27e52a ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 5a2bf84 ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 1680f21 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py f475798 ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 Diff: https://reviews.apache.org/r/44210/diff/ Testing --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On April 29, 2016, 12:12 a.m., Alexander Denissov wrote: > > Do we have a branch cut with these changes in so that we can test HAWQ and > > PXF with this new logic ? After testing the branch can be merged to trunk > > and we will avoid any major surprises. > > Tim Thorpe wrote: > There isn't currently a branch. Do we really need a branch, wouldn't it > be just as easy to just apply the patch over trunk and try it out? If you > think we need a branch then, I'll ask that one get created. > > Alexander Denissov wrote: > We will be testing this patch with HAWQ to see if it behaves as expected > and there are no regressions. Hi Alex, please let me know when you have validated the patch with HAWQ so I can get it pushed in (after I try it out again over trunk). Thanks - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review131020 ------- On April 29, 2016, 8:11 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 29, 2016, 8:11 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 356adb1 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 5a2bf84 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 1680f21 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > f475798 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 2080c52 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/ --- (Updated May 4, 2016, 2:17 p.m.) Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Changes --- Fixes issues with stack service's advisor class naming convention and with HAWQ and PXF common services inheritance. Bugs: AMBARI-15226 https://issues.apache.org/jira/browse/AMBARI-15226 Repository: ambari Description --- Currently the stack advisor is defined under each stack version such as HDP/2.3. The problem with this is that it restricts the services that can be added to the stack. If a custom service is to be added, they would need to modify the stack advisor. If the configuration recommendation and validation can be done at the service level then the custom service could just include their own recommendations and validations separately. Diffs (updated) - ambari-server/src/main/assemblies/server.xml e1a4919 ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java df65010 ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 00c8696 ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java ca1968e ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java 6c6fa91 ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 636de37 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 356adb1 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java b7e09a9 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java d27e52a ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 5a2bf84 ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 1680f21 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py f475798 ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 Diff: https://reviews.apache.org/r/44210/diff/ Testing --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/ --- (Updated May 5, 2016, 2:50 p.m.) Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Changes --- Final patch file attached, which was merged into latest trunk. Bugs: AMBARI-15226 https://issues.apache.org/jira/browse/AMBARI-15226 Repository: ambari Description --- Currently the stack advisor is defined under each stack version such as HDP/2.3. The problem with this is that it restricts the services that can be added to the stack. If a custom service is to be added, they would need to modify the stack advisor. If the configuration recommendation and validation can be done at the service level then the custom service could just include their own recommendations and validations separately. Diffs (updated) - ambari-server/src/main/assemblies/server.xml e1a4919 ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java df65010 ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 00c8696 ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java ca1968e ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java 6c6fa91 ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 636de37 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 356adb1 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java b7e09a9 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java d27e52a ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 5a2bf84 ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 1680f21 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py cf0990d ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py fcb5407 Diff: https://reviews.apache.org/r/44210/diff/ Testing --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On May 4, 2016, 9:37 p.m., Matt wrote: > > We have tested the following changes: > > - The logic for stack_advisor which was pulled into service_advisor works > > well. > > - Inheritance between stacks works well. We tested inheritance from > > common-services into stack 2.3, and then inheriting from 2.3 to stack 2.4 > > with overrides. > > > > Enhancements: > > - Factor out utility functions that are common for both stack_advisor and > > service_advisor. > > - Cache service_advisors to avoid loading service_advisor multiple times > > for the same service. > > - At the moment, the service advisor class name under the stack should be > > of the format **ServiceAdvisor**. > > It would be great if this class can have a different name in the > > service_advisor.py file and specify the file path and class name in > > metainfo.xml under that stack, so that the class gets loaded. Hi Matt, are you able to push the patch in? I don't have committer access. If not can someone else? Thanks - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review131747 --- On May 5, 2016, 2:50 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated May 5, 2016, 2:50 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 356adb1 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 5a2bf84 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 1680f21 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > cf0990d > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > fcb5407 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 47138: stackadvisor uses getHostsForSlaveComponent with wrong parameter name
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47138/#review132419 --- Ship it! Ship It! - Tim Thorpe On May 9, 2016, 11:43 p.m., Lav Jain wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47138/ > --- > > (Updated May 9, 2016, 11:43 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Goutam > Tadi, jun aoki, Matt, and Tim Thorpe. > > > Bugs: AMBARI-16378 > https://issues.apache.org/jira/browse/AMBARI-16378 > > > Repository: ambari > > > Description > --- > > ``` > def getHostsForSlaveComponent(self, services, hosts, component, hostsList, > hostsComponentsMap, freeHosts): > componentName = component["StackServiceComponents"]["component_name"] > > if component["StackServiceComponents"]["cardinality"] == "ALL": > return hostsList > > componentIsPopulated = self.isComponentHostsPopulated(component) > if componentIsPopulated: > return component["StackServiceComponents"]["hostnames"] > > hostsForComponent = [] > > if self.isSlaveComponent(component): > cardinality = str(component["StackServiceComponents"]["cardinality"]) > if self.isComponentUsingCardinalityForLayout(component) and cardinality: > # cardinality types: 1+, 1-2, 1 > ``` > > The correct parameter name is comonentName (instead of component) > > > Diffs > - > > ambari-server/src/main/resources/stacks/stack_advisor.py 00b9d79 > > Diff: https://reviews.apache.org/r/47138/diff/ > > > Testing > --- > > Tested manually > > > Thanks, > > Lav Jain > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On May 2, 2016, 11:07 p.m., Lav Jain wrote: > > ambari-server/src/main/resources/stacks/stack_advisor.py, line 644 > > <https://reviews.apache.org/r/44210/diff/6/?file=1367511#file1367511line644> > > > > This method is not being called because the corresponding method in > > 2.0.6 stack advisor is overriding it. Hi Lav, I don't see the 2.0.6 stack advisor overriding this method (def validateComponentLayout). It does override the method getComponentLayoutValidations which gets called by validateComponentLayout but in that method it calls super getComponentLayoutValidations first: def getComponentLayoutValidations(self, services, hosts): """Returns array of Validation objects about issues with hostnames components assigned to""" items = super(HDP206StackAdvisor, self).getComponentLayoutValidations(services, hosts) - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review131409 ----------- On April 29, 2016, 8:11 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated April 29, 2016, 8:11 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 356adb1 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 5a2bf84 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/properties.json eac0dbd > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 1680f21 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > f475798 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 2080c52 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/ --- (Updated May 3, 2016, 2:21 p.m.) Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Changes --- This patch fixes the common-services naming convention so that it uses service name and version. I attempted to install HAWQ on a 3 node cluster. It recommended the standby on node 2 and the master on node 3. So looks like its working properly. Also I checked the services.json in /var/run/ambari-server/stack-recommendations/1 and it returns HAWQ200ServiceAdvisor. cat services.json | grep -A 2 -B 2 advisor_name "stack_name" : "HDP", "stack_version" : "2.3", "advisor_name" : "HAWQ200ServiceAdvisor", "advisor_path" : "/var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/service_advisor.py" }, Bugs: AMBARI-15226 https://issues.apache.org/jira/browse/AMBARI-15226 Repository: ambari Description --- Currently the stack advisor is defined under each stack version such as HDP/2.3. The problem with this is that it restricts the services that can be added to the stack. If a custom service is to be added, they would need to modify the stack advisor. If the configuration recommendation and validation can be done at the service level then the custom service could just include their own recommendations and validations separately. Diffs (updated) - ambari-server/src/main/assemblies/server.xml e1a4919 ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java df65010 ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 00c8696 ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java ca1968e ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java 6c6fa91 ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java 636de37 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java 356adb1 ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java b7e09a9 ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java d27e52a ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 5a2bf84 ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 1680f21 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py f475798 ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 Diff: https://reviews.apache.org/r/44210/diff/ Testing --- Ran all the non java unit tests. Total run:945 Total errors:0 Total failures:0 Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure their service advisors were called. Thanks, Tim Thorpe
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On May 3, 2016, 11:09 p.m., Matt wrote:
> > The functionality of ServiceAdvisor works as expected. However more
> > clarification is needed on how to use inheritance and override service
> > advisor defined in common-services.
> > For example:
> > HAWQ200ServiceAdvisor is defined under commmon-services. I'd like to
> > override the method getComponentLayoutScheme with some new logic for HDP
> > 2.3 stack. Followed the documentation in
> > ambari-server/src/main/resources/stacks/service_advisor.py but it did not
> > work as expected.
>
> Tim Thorpe wrote:
> Thanks Matt, for finding that issue. There were actually 2 problems.
> The first was with the advisor class naming convention for stack service's
> that contained the service_advisor.py rather than inherit it from
> common-services. It missed the stack version number in the advisor class
> name. The second issue was with the HAWQ and PXF service_advisor.py files
> which added to the stacks directory to the system path to load the
> stacks/service_advisor.py (for the default service advisor). I have modified
> that code to use the same dynamic loading I recommended for service advisor
> inheritance. In other words, I changed it to this:
>
> SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
> STACKS_DIR = os.path.join(SCRIPT_DIR, '../../../stacks/')
> PARENT_FILE = os.path.join(STACKS_DIR, 'service_advisor.py')
>
> try:
> with open(PARENT_FILE, 'rb') as fp:
> service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE,
> ('.py', 'rb', imp.PY_SOURCE))
> except Exception as e:
> traceback.print_exc()
> print "Failed to load parent"
>
> class HAWQ200ServiceAdvisor(service_advisor.ServiceAdvisor):
>
> I presume the reason this change is required is due to a naming conflict
> because all the files are called service_advisor.py.
I have also added a comment in stacks/service_advisor.py to look at the HAWQ
and PXF service advisors for examples on how to create your own service advisor.
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131579
---
On May 3, 2016, 7:04 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
>
> (Updated May 3, 2016, 7:04 p.m.)
>
>
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
>
>
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the stack advisor is defined under each stack version such as
> HDP/2.3. The problem with this is that it restricts the services that can be
> added to the stack. If a custom service is to be added, they would need to
> modify the stack advisor. If the configuration recommendation and validation
> can be done at the service level then the custom service could just include
> their own recommendations and validations separately.
>
>
> Diffs
> -
>
> ambari-server/src/main/assemblies/server.xml e1a4919
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
> df65010
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
> 00c8696
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
> ca1968e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
> 6c6fa91
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 636de37
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> 356adb1
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> b7e09a9
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> d27e52a
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> 5a2bf84
>
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
> PRE-CREA
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On May 3, 2016, 11:09 p.m., Matt wrote:
> > The functionality of ServiceAdvisor works as expected. However more
> > clarification is needed on how to use inheritance and override service
> > advisor defined in common-services.
> > For example:
> > HAWQ200ServiceAdvisor is defined under commmon-services. I'd like to
> > override the method getComponentLayoutScheme with some new logic for HDP
> > 2.3 stack. Followed the documentation in
> > ambari-server/src/main/resources/stacks/service_advisor.py but it did not
> > work as expected.
Thanks Matt, for finding that issue. There were actually 2 problems. The
first was with the advisor class naming convention for stack service's that
contained the service_advisor.py rather than inherit it from common-services.
It missed the stack version number in the advisor class name. The second issue
was with the HAWQ and PXF service_advisor.py files which added to the stacks
directory to the system path to load the stacks/service_advisor.py (for the
default service advisor). I have modified that code to use the same dynamic
loading I recommended for service advisor inheritance. In other words, I
changed it to this:
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
STACKS_DIR = os.path.join(SCRIPT_DIR, '../../../stacks/')
PARENT_FILE = os.path.join(STACKS_DIR, 'service_advisor.py')
try:
with open(PARENT_FILE, 'rb') as fp:
service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE,
('.py', 'rb', imp.PY_SOURCE))
except Exception as e:
traceback.print_exc()
print "Failed to load parent"
class HAWQ200ServiceAdvisor(service_advisor.ServiceAdvisor):
I presume the reason this change is required is due to a naming conflict
because all the files are called service_advisor.py.
- Tim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131579
-----------
On May 3, 2016, 7:04 p.m., Tim Thorpe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
>
> (Updated May 3, 2016, 7:04 p.m.)
>
>
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
>
>
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Currently the stack advisor is defined under each stack version such as
> HDP/2.3. The problem with this is that it restricts the services that can be
> added to the stack. If a custom service is to be added, they would need to
> modify the stack advisor. If the configuration recommendation and validation
> can be done at the service level then the custom service could just include
> their own recommendations and validations separately.
>
>
> Diffs
> -
>
> ambari-server/src/main/assemblies/server.xml e1a4919
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
> df65010
>
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
> 00c8696
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
> ca1968e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
> 6c6fa91
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
> 636de37
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
> 356adb1
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
> b7e09a9
>
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
> d27e52a
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java
> 5a2bf84
>
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py
> PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py
> 1680f21
> ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py
> f475798
> ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION
> ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e
> ambari-server/src/test
Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services
> On May 2, 2016, 11:07 p.m., Lav Jain wrote: > > ambari-server/src/main/resources/stacks/stack_advisor.py, line 644 > > <https://reviews.apache.org/r/44210/diff/6/?file=1367511#file1367511line644> > > > > This method is not being called because the corresponding method in > > 2.0.6 stack advisor is overriding it. > > Tim Thorpe wrote: > Hi Lav, I don't see the 2.0.6 stack advisor overriding this method (def > validateComponentLayout). It does override the method > getComponentLayoutValidations which gets called by validateComponentLayout > but in that method it calls super getComponentLayoutValidations first: > def getComponentLayoutValidations(self, services, hosts): > """Returns array of Validation objects about issues with hostnames > components assigned to""" > items = super(HDP206StackAdvisor, > self).getComponentLayoutValidations(services, hosts) > > Lav Jain wrote: > Hi Tim, > > The method being overridden is getConfigurationsValidationItems(): > > > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py#L929-L956 Thanks, just confirmed it. I'll make sure the 2.0.6 stack advisor calls the super method. - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44210/#review131409 --- On May 3, 2016, 2:21 p.m., Tim Thorpe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44210/ > --- > > (Updated May 3, 2016, 2:21 p.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. > > > Bugs: AMBARI-15226 > https://issues.apache.org/jira/browse/AMBARI-15226 > > > Repository: ambari > > > Description > --- > > Currently the stack advisor is defined under each stack version such as > HDP/2.3. The problem with this is that it restricts the services that can be > added to the stack. If a custom service is to be added, they would need to > modify the stack advisor. If the configuration recommendation and validation > can be done at the service level then the custom service could just include > their own recommendations and validations separately. > > > Diffs > - > > ambari-server/src/main/assemblies/server.xml e1a4919 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > df65010 > > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java > 00c8696 > > ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java > ca1968e > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java > 6c6fa91 > > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java > 636de37 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java > 356adb1 > > ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java > b7e09a9 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > d27e52a > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 5a2bf84 > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > PRE-CREATION > > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 1680f21 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > f475798 > ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION > ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 2080c52 > > Diff: https://reviews.apache.org/r/44210/diff/ > > > Testing > --- > > Ran all the non java unit tests. > > Total run:945 > Total errors:0 > Total failures:0 > > Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure > their service advisors were called. > > > Thanks, > > Tim Thorpe > >
Re: Review Request 50392: AMBARI-17856: quicklink url protocol type check should allow type specified as HTTP and HTTPS
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50392/#review143377 --- Ship it! Ship It! - Tim Thorpe On July 25, 2016, 12:40 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50392/ > --- > > (Updated July 25, 2016, 12:40 p.m.) > > > Review request for Ambari and Tim Thorpe. > > > Bugs: AMBARI-17856 > https://issues.apache.org/jira/browse/AMBARI-17856 > > > Repository: ambari > > > Description > --- > > quicklink url protocol type check should allow type specified as HTTP and > HTTPS in order to have the types be consistent (case-wise) with the other two > possible cases: HTTPS_ONLY and HTTP_ONLY > > > Diffs > - > > ambari-web/app/views/common/quick_view_link_view.js 17d7c04 > ambari-web/test/views/common/quick_link_view_test.js f9a52b1 > > Diff: https://reviews.apache.org/r/50392/diff/ > > > Testing > --- > > updated unit tests > > > Thanks, > > Di Li > >
Re: Review Request 50350: Config changes for Atlas in HDP 2.5 related to atlas.rest.address, atlas.cluster.name, etc
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50350/#review143541 --- Ship it! Ship It! - Tim Thorpe On July 26, 2016, 1:22 a.m., Alejandro Fernandez wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50350/ > --- > > (Updated July 26, 2016, 1:22 a.m.) > > > Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Madhan > Neethiraj, Nate Cole, Sumit Mohanty, Swapan Shridhar, Suma Shivaprasad, Sid > Wagle, and Tim Thorpe. > > > Bugs: AMBARI-17782 > https://issues.apache.org/jira/browse/AMBARI-17782 > > > Repository: ambari > > > Description > --- > > To support Atlas in HDP 2.5, make several config changes: > > /etc/hive/conf/atlas-application.properties: > > * atlas.rest.address (add this property) > > /etc/hive/conf/hive-site.xml: > > * atlas.cluster.name (remove this property) > * atlas.hook.hive.maxThreads (remove this property) > * atlas.hook.hive.minThreads (remove this property) > * atlas.rest.address (remove this property) > > /etc/storm/conf/atlas-application.properties: > > * atlas.rest.address (add this property) > > /etc/storm/conf/storm.yaml: > > * atlas.cluster.name (remove this property) > > /etc/falcon/conf/atlas-application.properties: > > * atlas.rest.address (add this property) > * atlas.cluster.name (this value is empty; need to set to correct value) > > > /etc/sqoop/conf/atlas-application.properties: > > * atlas.jaas.KafkaClient.option.keyTab (remove this property) > * atlas.jaas.KafkaClient.option.principal (remove this property) > * atlas.jaas.KafkaClient.option.storeKey (remove this property) > * atlas.jaas.KafkaClient.option.useKeyTab (remove this property) > * atlas.jaas.KafkaClient.option.useTicketCache=true (add this property) > * atlas.jaas.KafkaClient.option.renewTicket=true (add this property) > * atlas.rest.address (add this property) > > /etc/sqoop/conf/sqoop-site.xml: > > * atlas.cluster.name (remove this property) > > Also, there is no 'Custom sqoop-atlas-application.properties' section in Sqoop > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py > a384f69 > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py > f0a3f32 > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/application-properties.xml > 4305965 > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py > 6111c34 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml > 0590244 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml > c53dd39 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml > ec14979 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py > f53625c > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py > 6b05134 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py > 4f38055 > > ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml > d78c5be > > ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py > 963d169 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py > d765ca3 > > ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml > 3a3879b > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 8d5cdc9 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml > bfdb3d3 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml > a1b93e3 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 96b1400 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml > b2cc1c4 > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml > 86e0964 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 2099958 >
Re: Review Request 50539: EU to HDP 2.5 failed since config type 'sqoop-atlas-application.properties' has not been created
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50539/#review143873 --- Ship it! Ship It! - Tim Thorpe On July 27, 2016, 11:06 p.m., Alejandro Fernandez wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50539/ > --- > > (Updated July 27, 2016, 11:06 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, > Jonathan Hurley, Nate Cole, Sumit Mohanty, and Tim Thorpe. > > > Bugs: AMBARI-17935 > https://issues.apache.org/jira/browse/AMBARI-17935 > > > Repository: ambari > > > Description > --- > > STR > 1. Deploy Ambari 2.2.2 with HDP 2.4 and Sqoop > 2. Kerberize the cluster > 3. Upgrade Ambari to 2.4.0 > 4. Install bits for HDP 2.5 and attempt to perform Express Upgrade > > Observed error during Express Upgrade > ``` > Failed on: Updating configuration sqoop-atlas-application.properties > > Server action failed > ``` > > This happens because the sqoop-atlas-application.properties config in HDP 2.5 > hasn't been created yet since it doesn't actually have any configs. To fix > it, add some configs so the merging process creates the new config type. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java > f7de8a9 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml > 133db26 > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml > d648638 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/configuration/sqoop-atlas-application.properties.xml > PRE-CREATION > > Diff: https://reviews.apache.org/r/50539/diff/ > > > Testing > --- > > Verified during EU from HDP 2.4 to 2.5 with Sqoop. > > Python unit tests passed, > > -- > Total run:1034 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Alejandro Fernandez > >
Re: Review Request 50539: EU to HDP 2.5 failed since config type 'sqoop-atlas-application.properties' has not been created
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50539/#review143872 --- ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java (line 243) <https://reviews.apache.org/r/50539/#comment209827> Spelling error "finnd" - Tim Thorpe On July 27, 2016, 11:06 p.m., Alejandro Fernandez wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50539/ > --- > > (Updated July 27, 2016, 11:06 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, > Jonathan Hurley, Nate Cole, Sumit Mohanty, and Tim Thorpe. > > > Bugs: AMBARI-17935 > https://issues.apache.org/jira/browse/AMBARI-17935 > > > Repository: ambari > > > Description > --- > > STR > 1. Deploy Ambari 2.2.2 with HDP 2.4 and Sqoop > 2. Kerberize the cluster > 3. Upgrade Ambari to 2.4.0 > 4. Install bits for HDP 2.5 and attempt to perform Express Upgrade > > Observed error during Express Upgrade > ``` > Failed on: Updating configuration sqoop-atlas-application.properties > > Server action failed > ``` > > This happens because the sqoop-atlas-application.properties config in HDP 2.5 > hasn't been created yet since it doesn't actually have any configs. To fix > it, add some configs so the merging process creates the new config type. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java > f7de8a9 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml > 133db26 > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml > d648638 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/configuration/sqoop-atlas-application.properties.xml > PRE-CREATION > > Diff: https://reviews.apache.org/r/50539/diff/ > > > Testing > --- > > Verified during EU from HDP 2.4 to 2.5 with Sqoop. > > Python unit tests passed, > > -- > Total run:1034 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Alejandro Fernandez > >
Re: Review Request 50087: Ambari should have a script to add new repository and service to existing stack
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50087/#review142730 --- I really like the ability to add new repos. I'm not so certain we need another way of adding services to an existing stack. Jayush and I already added new code to allow Ambari to add new stacks, extensions and custom services - see AMBARI_15663. https://issues.apache.org/jira/browse/AMBARI-15663 A custom service can be added via extension-definitions or stack-addon-service-definitions management packs. I understand that this mechanism is a simple and quick way to add HAWQ and PXF. Unfortunately this script only works for those custom services which are actually part of Ambari's common-services. Ideally the ability to add new repos would be done in a separate script to allow it to be used when adding extension-definitions or stack-addon-service-definitions management packs. - Tim Thorpe On July 18, 2016, 8:51 p.m., Lav Jain wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50087/ > --- > > (Updated July 18, 2016, 8:51 p.m.) > > > Review request for Ambari, Alexander Denissov, Jayush Luniya, Matt, and Tim > Thorpe. > > > Bugs: AMBARI-17717 > https://issues.apache.org/jira/browse/AMBARI-17717 > > > Repository: ambari > > > Description > --- > > Ambari should have a script that users can run to add a custom service and > repository to the stack or an existing cluster. > > ``` > Lavs-MacBook-Pro:scripts ljain$ ./add_common_service.py -h > Usage: add_common_service.py [options] > > Options: > -h, --helpshow this help message and exit > -u USER, --user=USER Ambari login username (Required) > -p PASSWORD, --password=PASSWORD > Ambari login password. Providing password through > command line is not recommended. The script prompts > for the password. > -t STACK, --stack=STACK > Stack Name and Version to be added (Required).(Eg: > HDP-2.4 or HDP-2.5) > -s SERVICE, --service=SERVICE > Service Name and Version to be added.(Eg: HAWQ/2.0.0 > or PXF/3.0.0) > -r REPOURL, --repourl=REPOURL > Repository URL which points to the rpm packages > -i REPOID, --repoid=REPOID > Repository ID of the new repository > -o OSTYPE, --ostype=OSTYPE > OS for the new repository (Eg: redhat6) > ``` > > > Diffs > - > > ambari-server/src/main/resources/scripts/add_common_service.py PRE-CREATION > > Diff: https://reviews.apache.org/r/50087/diff/ > > > Testing > --- > > Tested manually with various combinations > > > Thanks, > > Lav Jain > >
Review Request 50237: AMBARI-17810 - Extensions directory shouldn't be packaged in ambari-server RPM
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50237/ --- Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako. Bugs: AMBARI-17810 https://issues.apache.org/jira/browse/AMBARI-17810 Repository: ambari Description --- Currently the extensions directory is packaged in the ambari-server RPM. This will create a problem during upgrade. Any extensions which have been added in the extensions directory will be wiped out. This defeats the point of packaging them in a directory separate from the stacks directory. Diffs - ambari-server/src/main/assemblies/server.xml 2439e17 Diff: https://reviews.apache.org/r/50237/diff/ Testing --- mvn clean test mvnpackage Thanks, Tim Thorpe
Review Request 49328: AMBARI-17465 - Management packs should be able to install extensions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49328/ --- Review request for Ambari, Jayush Luniya, Mahadev Konar, and Sumit Mohanty. Bugs: AMBARI-17465 https://issues.apache.org/jira/browse/AMBARI-17465 Repository: ambari Description --- Currently management packs (AMBARI-14854) can only add stacks and addon services. Now that AMBARI-12885 has been resolved, the management packs should be able to add extensions as well. Diffs - ambari-server/src/main/python/ambari_server/serverConfiguration.py e868f96 ambari-server/src/main/python/ambari_server/setupMpacks.py 98811f5 Diff: https://reviews.apache.org/r/49328/diff/ Testing --- Manual testing ambari-server install-mpack --mpack=/root/mpacks/my-extension-1.0.0.0.tar.gz -v Thanks, Tim Thorpe
Re: Review Request 49521: Move service advisor tests for HAWQ and PXF
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49521/#review140754 --- Ship it! Ship It! - Tim Thorpe On July 4, 2016, 12:33 a.m., Lav Jain wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49521/ > --- > > (Updated July 4, 2016, 12:33 a.m.) > > > Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush > Luniya, Matt, and Tim Thorpe. > > > Bugs: AMBARI-16647 > https://issues.apache.org/jira/browse/AMBARI-16647 > > > Repository: ambari > > > Description > --- > > Currently the service advisor tests for HAWQ and PXF are under > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > Move the tests to the service specific service_advisor files: > ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py > ambari-server/src/test/python/common-services/PXF/test_service_advisor.py > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py > e254094 > ambari-server/src/main/resources/stacks/stack_advisor.py d8685c3 > ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py > PRE-CREATION > ambari-server/src/test/python/common-services/PXF/test_service_advisor.py > PRE-CREATION > ambari-server/src/test/python/common-services/configs/hawq_default.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-hawq-3-hosts.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-hawq-pxf-hdfs.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-master_ambari_colo-3-hosts.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-master_standby_colo-3-hosts.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-nohawq-3-hosts.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-normal-hawq-3-hosts.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-normal-nohawq-3-hosts.json > PRE-CREATION > > ambari-server/src/test/python/common-services/configs/services-standby_ambari_colo-3-hosts.json > PRE-CREATION > > ambari-server/src/test/python/stacks/2.3/HAWQ/test_alert_component_status.py > > ambari-server/src/test/python/stacks/2.3/HAWQ/test_alert_segment_registration_status.py > 6bb5930 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_alert_sync_status.py > fd4f474 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqmaster.py 88fb008 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqsegment.py 8639ca5 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqstandby.py b406723 > ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py > 8d97baa > ambari-server/src/test/python/stacks/2.3/HAWQ/test_utils.py > ambari-server/src/test/python/stacks/2.3/PXF/test_alerts_api_status.py > ambari-server/src/test/python/stacks/2.3/PXF/test_pxf.py 1147a7e > ambari-server/src/test/python/stacks/2.3/PXF/test_service_advisor.py > 4ea3bfb > ambari-server/src/test/python/stacks/2.3/common/hosts-1-host.json > ambari-server/src/test/python/stacks/2.3/common/hosts-3-hosts.json > ambari-server/src/test/python/stacks/2.3/common/services-hawq-1-host.json > 515ba7d > ambari-server/src/test/python/stacks/2.3/common/services-hawq-3-hosts.json > 515ba7d > ambari-server/src/test/python/stacks/2.3/common/services-hawq-pxf-hdfs.json > 0bf459d > > ambari-server/src/test/python/stacks/2.3/common/services-master_ambari_colo-3-hosts.json > 1657ccf > > ambari-server/src/test/python/stacks/2.3/common/services-master_standby_colo-3-hosts.json > cd5d02c > > ambari-server/src/test/python/stacks/2.3/common/services-nohawq-3-hosts.json > beeb62d > > ambari-server/src/test/python/stacks/2.3/common/services-normal-hawq-3-hosts.json > 5495d77 > > ambari-server/src/test/python/stacks/2.3/common/services-normal-nohawq-3-hosts.json > 2149317 > > ambari-server/src/test/python/stacks/2.3/common/services-standby_ambari_colo-3-hosts.json > 92a8e58 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 2944f6f > ambari-server/src/test/python/stacks/2.3/configs/hawq_default.json ebff461 > ambari-server/src/test/python/stacks/2.3/configs/pxf_default.json > > Diff: https://reviews.apache.org/r/49521/diff/ > > > Testing > --- > > Total run:1005 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Lav Jain > >
Re: Review Request 49328: AMBARI-17465 - Management packs should be able to install extensions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49328/ --- (Updated July 8, 2016, 2:43 p.m.) Review request for Ambari, Jayush Luniya, Mahadev Konar, and Sumit Mohanty. Changes --- Added and updated unit tests Bugs: AMBARI-17465 https://issues.apache.org/jira/browse/AMBARI-17465 Repository: ambari Description --- Currently management packs (AMBARI-14854) can only add stacks and addon services. Now that AMBARI-12885 has been resolved, the management packs should be able to add extensions as well. Diffs (updated) - ambari-server/conf/unix/ambari.properties abd72ad ambari-server/src/main/python/ambari_server/serverConfiguration.py e868f96 ambari-server/src/main/python/ambari_server/setupMpacks.py 98811f5 ambari-server/src/test/python/TestMpacks.py a7c6092 ambari-server/src/test/python/mpacks/myextension-ambari-mpack-1.0.0.0/extensions/MYEXTENSION/1.0/metainfo.xml PRE-CREATION ambari-server/src/test/python/mpacks/myextension-ambari-mpack-1.0.0.0/extensions/MYEXTENSION/1.1/metainfo.xml PRE-CREATION ambari-server/src/test/python/mpacks/myextension-ambari-mpack-1.0.0.0/mpack.json PRE-CREATION ambari-server/src/test/python/mpacks/myservice-ambari-mpack-1.0.0.0/mpack.json faade4a Diff: https://reviews.apache.org/r/49328/diff/ Testing --- Manual testing ambari-server install-mpack --mpack=/root/mpacks/hdp-ambari-mpack-1.0.0.0.tar.gz --purge -v ambari-server install-mpack --mpack=/root/mpacks/myservice-ambari-mpack-1.0.0.0.tar.gz -v ambari-server install-mpack --mpack=/root/mpacks/myservice2-ambari-mpack-1.0.0.0.tar.gz -v ambari-server install-mpack --mpack=/root/mpacks/myextension-1.0.0.0.tar.gz -v Thanks, Tim Thorpe
Review Request 49831: AMBARI-17562 - Adding single stack, extension and service should be removed from management pack support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49831/ --- Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Mahadev Konar, and Sumit Mohanty. Bugs: AMBARI-17562 https://issues.apache.org/jira/browse/AMBARI-17562 Repository: ambari Description --- This code is duplicated by the ability to add multiple stacks, extensions and addon services. There is no reason to maintain both code paths. Diffs - ambari-server/src/main/python/ambari_server/setupMpacks.py 5acc92c ambari-server/src/test/python/TestMpacks.py 0995280 ambari-server/src/test/python/mpacks/myservice-ambari-mpack-1.0.0.0/mpack.json 0af6949 Diff: https://reviews.apache.org/r/49831/diff/ Testing --- mvn clean test -DskipSurefireTests No errors related to the mpack tests Thanks, Tim Thorpe
Re: Review Request 49861: Changes to stack advisor framework to help with service advisors
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49861/#review141670 --- Ship it! Ship It! - Tim Thorpe On July 9, 2016, 9:22 p.m., Robert Levas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49861/ > --- > > (Updated July 9, 2016, 9:22 p.m.) > > > Review request for Ambari, Dmytro Sen, Jayush Luniya, Matt, Srimanth Gunturi, > and Tim Thorpe. > > > Bugs: AMBARI-17642 > https://issues.apache.org/jira/browse/AMBARI-17642 > > > Repository: ambari > > > Description > --- > > The following changes are needed in the stack advisor framework to help with > service advisors: > > - Add additional logging to show why a service advisor implementation was not > loaded > - Move `isSecurityEnabled` from `stacks/HDP/2.0.6/services/stack_advisor.py` > to a class member of `DefaultStackAdvisor` in `stacks/stack_advisor.py` so > that all stack and service advisors may be able to use it > > > Diffs > - > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 3f66216 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > 990c314 > ambari-server/src/main/resources/stacks/stack_advisor.py 23c0320 > > Diff: https://reviews.apache.org/r/49861/diff/ > > > Testing > --- > > Manually tested > > # Local test results (python only): > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2:56.308s > [INFO] Finished at: Sat Jul 09 16:12:31 EDT 2016 > [INFO] Final Memory: 65M/1700M > [INFO] > > > #Jenkins test results: PENDING > > > Thanks, > > Robert Levas > >
Re: Review Request 49328: AMBARI-17465 - Management packs should be able to install extensions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49328/ --- (Updated June 30, 2016, 1:38 p.m.) Review request for Ambari, Jayush Luniya, Mahadev Konar, and Sumit Mohanty. Changes --- Changed testing to include regression tests from original mpacks JIRA (AMBARI-15663) Bugs: AMBARI-17465 https://issues.apache.org/jira/browse/AMBARI-17465 Repository: ambari Description --- Currently management packs (AMBARI-14854) can only add stacks and addon services. Now that AMBARI-12885 has been resolved, the management packs should be able to add extensions as well. Diffs - ambari-server/src/main/python/ambari_server/serverConfiguration.py e868f96 ambari-server/src/main/python/ambari_server/setupMpacks.py 98811f5 Diff: https://reviews.apache.org/r/49328/diff/ Testing (updated) --- Manual testing ambari-server install-mpack --mpack=/root/mpacks/hdp-ambari-mpack-1.0.0.0.tar.gz --purge -v ambari-server install-mpack --mpack=/root/mpacks/myservice-ambari-mpack-1.0.0.0.tar.gz -v ambari-server install-mpack --mpack=/root/mpacks/myservice2-ambari-mpack-1.0.0.0.tar.gz -v ambari-server install-mpack --mpack=/root/mpacks/myextension-1.0.0.0.tar.gz -v Thanks, Tim Thorpe
Re: Review Request 49328: AMBARI-17465 - Management packs should be able to install extensions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49328/ --- (Updated June 30, 2016, 5:16 p.m.) Review request for Ambari, Jayush Luniya, Mahadev Konar, and Sumit Mohanty. Changes --- Added more comments to the extension functions Bugs: AMBARI-17465 https://issues.apache.org/jira/browse/AMBARI-17465 Repository: ambari Description --- Currently management packs (AMBARI-14854) can only add stacks and addon services. Now that AMBARI-12885 has been resolved, the management packs should be able to add extensions as well. Diffs (updated) - ambari-server/src/main/python/ambari_server/serverConfiguration.py e868f96 ambari-server/src/main/python/ambari_server/setupMpacks.py 98811f5 Diff: https://reviews.apache.org/r/49328/diff/ Testing --- Manual testing ambari-server install-mpack --mpack=/root/mpacks/hdp-ambari-mpack-1.0.0.0.tar.gz --purge -v ambari-server install-mpack --mpack=/root/mpacks/myservice-ambari-mpack-1.0.0.0.tar.gz -v ambari-server install-mpack --mpack=/root/mpacks/myservice2-ambari-mpack-1.0.0.0.tar.gz -v ambari-server install-mpack --mpack=/root/mpacks/myextension-1.0.0.0.tar.gz -v Thanks, Tim Thorpe
Re: Review Request 55848: AMBARI-19630: Ambari should accept stack version in format of x.x.x.x without the build level digits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55848/#review163121 --- Ship it! Ship It! - Tim Thorpe On Jan. 25, 2017, 9:36 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55848/ > --- > > (Updated Jan. 25, 2017, 9:36 p.m.) > > > Review request for Ambari, Jayush Luniya and Tim Thorpe. > > > Bugs: AMBARI-19630 > https://issues.apache.org/jira/browse/AMBARI-19630 > > > Repository: ambari > > > Description > --- > > Ambari current requires stack version ( used in the repo) as x.x.x.x-, > Ambari should be flexible enough to accept both x.x.x.x- and x.x.x.x > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py > 7274a59 > > ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py > 615a0cd > ambari-server/src/test/python/TestVersionSelectUtil.py 38798e2 > > Diff: https://reviews.apache.org/r/55848/diff/ > > > Testing > --- > > unit tests > build an trunk cluster with version 2.6.0.0, run upgrade, downgrade, then > upgrade again, at the Finalize step verify the hostcomponentstate table has > 2.6.0.0 as the version and upgrade_state as COMPLETE instead of the previous > "IN_PROGESS". then be able to click the finalize and finalize the upgrade. > > > Thanks, > > Di Li > >
Re: Review Request 55960: AMBARI-19715: HostCleanup remove ambari.repo when ambari.repo has repo ID that doesn't begin with word AMBARI
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55960/#review163122 --- Ship it! Ship It! - Tim Thorpe On Jan. 25, 2017, 9:59 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55960/ > --- > > (Updated Jan. 25, 2017, 9:59 p.m.) > > > Review request for Ambari and Tim Thorpe. > > > Bugs: AMBARI-19715 > https://issues.apache.org/jira/browse/AMBARI-19715 > > > Repository: ambari > > > Description > --- > > HostCleanup remove ambari.repo when ambari.repo has repo ID that doesn't > begin with word AMBARI > > > Diffs > - > > ambari-agent/src/test/python/resource_management/TestPackagesAnalyzer.py > f898919 > > ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py > 9df2b64 > > Diff: https://reviews.apache.org/r/55960/diff/ > > > Testing > --- > > unit testing > run hostcheck custom action, verify the hostcheck_custom_action.result > doesn't contain customely named Ambari repo, I used GITHUB_AMBARI as my > ambari repo id. > > > Thanks, > > Di Li > >
Re: Review Request 55774: AMBARI-19636: Provide default values for Kafka nofile and nproc limit properties
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55774/#review162651 --- Ship it! Ship It! - Tim Thorpe On Jan. 20, 2017, 4:01 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55774/ > --- > > (Updated Jan. 20, 2017, 4:01 p.m.) > > > Review request for Ambari and Tim Thorpe. > > > Bugs: AMBARI-19636 > https://issues.apache.org/jira/browse/AMBARI-19636 > > > Repository: ambari > > > Description > --- > > Provide default values for Kafka nofile and nproc limit properties > > > Diffs > - > > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py > 6c7ff69 > > Diff: https://reviews.apache.org/r/55774/diff/ > > > Testing > --- > > existing unit tests > build ambari rpms with trunk code, install a trunk cluster. > > > Thanks, > > Di Li > >
Re: Review Request 55848: AMBARI-19630: Ambari should accept stack version in format of x.x.x.x without the build level digits
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55848/#review162654 --- Ship it! Ship It! - Tim Thorpe On Jan. 23, 2017, 4:51 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55848/ > --- > > (Updated Jan. 23, 2017, 4:51 p.m.) > > > Review request for Ambari, Jayush Luniya and Tim Thorpe. > > > Bugs: AMBARI-19630 > https://issues.apache.org/jira/browse/AMBARI-19630 > > > Repository: ambari > > > Description > --- > > Ambari current requires stack version ( used in the repo) as x.x.x.x-, > Ambari should be flexible enough to accept both x.x.x.x- and x.x.x.x > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py > 7274a59 > > ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py > 615a0cd > ambari-server/src/test/python/TestVersionSelectUtil.py 38798e2 > > Diff: https://reviews.apache.org/r/55848/diff/ > > > Testing > --- > > unit tests > build an trunk cluster with version 2.6.0.0, run upgrade, downgrade, then > upgrade again, at the Finalize step verify the hostcomponentstate table has > 2.6.0.0 as the version and upgrade_state as COMPLETE instead of the previous > "IN_PROGESS". then be able to click the finalize and finalize the upgrade. > > > Thanks, > > Di Li > >
Re: Review Request 55847: AMBARI-19657: Downgrade button does not work after restart Ambari server when upgrade wizard was left open
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55847/#review162667 --- Ship it! Ship It! - Tim Thorpe On Jan. 23, 2017, 6:05 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55847/ > --- > > (Updated Jan. 23, 2017, 6:05 p.m.) > > > Review request for Ambari, Sangeeta Ravindran and Tim Thorpe. > > > Bugs: AMBARI-19657 > https://issues.apache.org/jira/browse/AMBARI-19657 > > > Repository: ambari > > > Description > --- > > Run express/rolling upgrade on a trunk cluster. While the upgrade wizard UI > pauses on a manual step or a failed step, restart ambari server. Once the > Ambari Server restarts, reload Ambari web UI, observe that the wizard UI pops > up automatically with the paused step showing the Downgrade button. Clicking > the downgrade button doesn't do anything in this case because the > "currentVersion" variable in the controller isn't set yet. Workaround is to > close the wizard to force the Stack/Version page to reload, then reopen the > wizard and click the Downgrade button. > > > Diffs > - > > ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js > 4f88d2f > > Diff: https://reviews.apache.org/r/55847/diff/ > > > Testing > --- > > build trunk code, run upgrade, restart ambari server when the upgrade wizard > is opened and paused at a menu step. > existing web UI unit tests. > > > Thanks, > > Di Li > >
Re: Review Request 55584: AMBARI-19537: Provide default value for yarn leveldb state store path
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55584/#review161763 --- Ship it! Ship It! - Tim Thorpe On Jan. 16, 2017, 6:12 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55584/ > --- > > (Updated Jan. 16, 2017, 6:12 p.m.) > > > Review request for Ambari and Tim Thorpe. > > > Bugs: AMBARI-19537 > https://issues.apache.org/jira/browse/AMBARI-19537 > > > Repository: ambari > > > Description > --- > > Provide default value for yarn leveldb state store path > > > Diffs > - > > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py > 2fb7bff > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py > 23a25a0 > > Diff: https://reviews.apache.org/r/55584/diff/ > > > Testing > --- > > existing unit tests > build rpms off trunk source code and deploy a cluster. > > > Thanks, > > Di Li > >
Re: Review Request 55720: AMBARI-19615 clearer error messages for stack_select.py when a role doesn't have a select component name
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55720/#review162298 --- Ship it! Ship It! - Tim Thorpe On Jan. 19, 2017, 3:46 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55720/ > --- > > (Updated Jan. 19, 2017, 3:46 p.m.) > > > Review request for Ambari and Tim Thorpe. > > > Bugs: AMBARI-19615 > https://issues.apache.org/jira/browse/AMBARI-19615 > > > Repository: ambari > > > Description > --- > > clearer error messages for stack_select.py when a role doesn't have a select > component name > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py > 4f6ccd2 > > Diff: https://reviews.apache.org/r/55720/diff/ > > > Testing > --- > > existing unit test > patch trunk code build ambari rpms and install a trunk cluster > > > Thanks, > > Di Li > >
Re: Review Request 57146: AMBARI-20221 Ambari db schema update should be able to compare Ambari versions with build text
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57146/#review167101 --- Ship it! Ship It! - Tim Thorpe On Feb. 28, 2017, 1:34 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57146/ > --- > > (Updated Feb. 28, 2017, 1:34 p.m.) > > > Review request for Ambari and Tim Thorpe. > > > Bugs: AMBARI-20221 > https://issues.apache.org/jira/browse/AMBARI-20221 > > > Repository: ambari > > > Description > --- > > SchemaUpgradeHelper should be able to handle Ambari with versions like > 1.2.3_mycompany_00 instead of just versions like 1.2.3.4 > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/utils/VersionUtils.java > d3d8592 > > ambari-server/src/test/java/org/apache/ambari/server/utils/TestVersionUtils.java > 821565e > > Diff: https://reviews.apache.org/r/57146/diff/ > > > Testing > --- > > existing unit tests, new unit tests, trunk cluster install > > > Thanks, > > Di Li > >
Re: Review Request 51604: AMBARI-11639: Ambari Admin View URL does not get properly parsed for custom versions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51604/#review147668 --- Ship it! Ship It! - Tim Thorpe On Sept. 2, 2016, 3:26 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51604/ > --- > > (Updated Sept. 2, 2016, 3:26 p.m.) > > > Review request for Ambari, Jesus Alvarez and Tim Thorpe. > > > Bugs: AMBARI-11639 > https://issues.apache.org/jira/browse/AMBARI-11639 > > > Repository: ambari > > > Description > --- > > When building a custom version of ADMIN VIEW jar with new version such as > "2.1.0_custom", the UI returns a 404 message upon redirect. > > Upon loading ambari-server (http://node1.bigdata:8080) for the first time, > the URL is redirected to: > http://node1.bigdata:8080/views/ADMIN_VIEW/2.1.0_custom/INSTANCE/# , which > returns a 404 missing page. > > If the URL is manually changed to > http://node1.bigdata:8080/views/ADMIN_VIEW/2.1.0/INSTANCE/# , the UI loads – > though links in ambari UI remain broken. > > This appears to be caused by > latestVersion = sortedMappedVersions[sortedMappedVersions.length-1]; > and ambari.service.load_server_version ; each parsing version string > differently. > > > Diffs > - > > ambari-web/app/router.js d367ce3 > ambari-web/app/views/main/admin/stack_upgrade/versions_view.js 59baf07 > ambari-web/test/router_test.js e17c9e2 > ambari-web/test/views/main/admin/stack_upgrade/version_view_test.js b7df818 > > Diff: https://reviews.apache.org/r/51604/diff/ > > > Testing > --- > > existing unit test > added new unit test > > > Thanks, > > Di Li > >
Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51238/#review148002 --- ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java (line 1129) <https://reviews.apache.org/r/51238/#comment215353> This won't include extension services. You would need to loop through all the ServiceInfo objects and check to see if they have repos. - Tim Thorpe On Sept. 5, 2016, 1:26 p.m., Balázs Bence Sári wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51238/ > --- > > (Updated Sept. 5, 2016, 1:26 p.m.) > > > Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and > Sebastian Toader. > > > Bugs: AMBARI-15538 > https://issues.apache.org/jira/browse/AMBARI-15538 > > > Repository: ambari > > > Description > --- > > Implementation contains the following things: > - On startup, add-on service repos are loaded into the stack model > - On startup, add-on service repos are merged into the VDF's downloaded by > LatestRepoCallable > - On startup, if the is an existing cluster, it's repository version entity > is potentially updated with new add-on service repos. > - Repository definitions contain two new optional fields: service_name and > service_version. (null for stack repositories) > - Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5) > - New unit tests > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java > 9f7419c > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > b1fd592 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java > d20b1d7 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java > 02fc2ec > ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > 3acc617 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 23b0218 > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 7bcd08b > > ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java > 29776ed > ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java > 14ff9de > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java > 50d6028 > ambari-server/src/main/resources/version_definition.xsd bd49028 > > ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java > 1d73ff3 > > ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java > PRE-CREATION > > ambari-server/src/test/resources/common-services/ADDON/1.0/configuration/addon-env.xml > PRE-CREATION > ambari-server/src/test/resources/common-services/ADDON/1.0/metainfo.xml > PRE-CREATION > > ambari-server/src/test/resources/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest_initialRepos.json > PRE-CREATION > > ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/metainfo.xml > PRE-CREATION > > ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/repos/repoinfo.xml > PRE-CREATION > > contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/configuration/microsoft-r-env.xml > PRE-CREATION > > contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/microsoft_r.py > 61ea96b > > contrib/management-packs/microsoft-r_mpack/src/main/resources/custom-services/MICROSOFT_R/8.0.0/repos/repoinfo.xml > PRE-CREATION > contrib/management-packs/microsoft-r_mpack/src/main/resources/mpack.json > f90ccce > > Diff: https://reviews.apache.org/r/51238/diff/ > > > Testing > --- > > - Manually tested > - Wrote new unit tests > - All unit tests passed except two which were failing in CI builds as well. > > > Thanks, > > Balázs Bence Sári > >
Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51238/#review147557 --- ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java (line 138) <https://reviews.apache.org/r/51238/#comment214750> The Optional class is only in JDK8. Currently Ambari is supposed to work with both JDK7 and 8. - Tim Thorpe On Aug. 26, 2016, 9:25 a.m., Balázs Bence Sári wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51238/ > --- > > (Updated Aug. 26, 2016, 9:25 a.m.) > > > Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and > Sebastian Toader. > > > Bugs: AMBARI-15538 > https://issues.apache.org/jira/browse/AMBARI-15538 > > > Repository: ambari > > > Description > --- > > Implementation contains the following things: > - On startup, add-on service repos are loaded into the stack model > - On startup, add-on service repos are merged into the VDF's downloaded by > LatestRepoCallable > - On startup, if the is an existing cluster, it's repository version entity > is potentially updated with new add-on service repos. > - Repository definitions contain two new optional fields: service_name and > service_version. (null for stack repositories) > - Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5) > - New unit tests > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > 92d47df > > ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java > 36a2d99 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > 63c99c6 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java > 097f01c > > ambari-server/src/main/java/org/apache/ambari/server/controller/RepositoryResponse.java > 30bd0db > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RepositoryResourceProvider.java > 3b5b0a7 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java > 02fc2ec > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryEntity.java > 49d53a9 > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java > 25aa62b > ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java > bfba021 > ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java > 0606f2a > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java > 7bcd08b > > ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java > 29776ed > ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java > 14ff9de > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/LatestRepoCallable.java > 3c7c001 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepositoryXml.java > 4a0ae3b > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java > 50d6028 > > ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java > 3b5ed28 > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java > d806dde > ambari-server/src/main/resources/version_definition.xsd bd49028 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RepositoryResourceProviderTest.java > 3a7b19b > > ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java > 1d73ff3 > > ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache
Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services
> On Aug. 29, 2016, 5:27 p.m., Nate Cole wrote: > > I was under the impression that applying an m-pack would DIRECTLY update > > repoinfo.xml on the filesystem. Has the design changed since then? > > > > What is the purpose of carrying service version around with the repo? When > > updating m-packs, that just means that there's that much more work to do to > > get it right. > > > > How are you going to prevent an m-pack from overwriting an existing repo? > > (What if an m-pack names a repo "HDP-UTILS" or something - then they just > > blew over ours)? > > > > You need to fix the code formatting so it's consistent and a bit easier to > > read. > > > > In general, I don't like the idea of using service name to tie a repo to a > > service. Maybe name this "reference" or a different indicator of where > > this repo came from. > > Jayush Luniya wrote: > RE: I was under the impression that applying an m-pack would DIRECTLY > update repoinfo.xml on the filesystem. > Personally, it would be better if we can avoid updates to the stack's > repoinfo.xml and have the support to define repos at service level. > > Balázs Bence Sári wrote: > My implementation is based on this comment to the Jira (by Srimanth): > > "Discussed with Nate Cole and Jayush Luniya, and the approach for > custom-services to specify their own repo location will be to provide a > /repos/repoinfo.xml inside the stack-version they will be in. This repo file > will be loaded by Ambari during startup into the > /api/v1/stacks/HDP/versions/2.4/repository_versions repos. Service repo files > have a restriction that their (repo-name, base-url) locations should be > unique and not conflict. When conflicts do occur, they will not be loaded > into the stacks model. > Now the management-pack will provide such repos/ folder in > mpacks/custom-services/8.0.0/repos which will be linked into the stacks/ > folder. > > ambari/ambari-server/src/main/resources/stacks/HDP/2.3/services/SERVICE_NAME/repos > -> mpacks/custom-services/8.0.0/repos" > > Overwriting a stack repo is prohibited in the implementation. > > service_name and service_version come from Srimanth, I can rename or > discard them. What should happen with them? Should I delete service_version > and rename service_name to reference? My two cents are that it's worth > keeping one to indicate that the repo does not come with the stack. > > Nate Cole wrote: > Association should be with the service itself (I think that's what Jayush > is saying), and we can access those as part of the stack endpoint for the > service, but should not be integrated with the stack definition. We should > then include that repo info when we create the tasks to agents. > > Balázs Bence Sári wrote: > I think that's a bit different than how it was in the Jira comment (at > least to my interpetation) which I looked like a result of consensus. I can > implement it however. > > Also, we also need to make sure the user can overwrite those repo url's > when installing a cluster (Microsoft wants no public repo, repo locatinon in > the final mpack will be empty, corporates are expected to create their > private repos). With the current implementation, > api/v1/stacks/HDP/versions/2.3/operating_systems/redhat6 will return the > service repo along the stack repos and the user can override it on the UI. If > I move the repo to the service, should this API call still return it? > (guessing that UI uses this API to get the repos). > > Nate Cole wrote: > We can aggregate them in the API response, but I don't know that they > should be stored with the stack repos. I added stack extensions (https://cwiki.apache.org/confluence/display/AMBARI/Extensions) for the purpose of allowing shared resources for a collection of custom services. The way you are implementing service level repositories seems wrong. If you have multiple services that require a custom repository, then you are randomly selecting which service's repoinfo.xml you will use. If it really is a shared repository, it should be included at the extension level. In general, I think we should be encouraging third parties to deliver their custom services in a consistent way (extensions installed with mpacks). - Tim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51238/#review147153 --- On Aug. 26, 2016, 9:25 a.m., Balázs Bence Sári wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51238/ > --- > > (Updated Aug. 26, 2016, 9:25 a.m.) > > > Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and > Sebastian Toader. >
Review Request 51761: AMBARI-18050 - Upgrade pre-req check code needs to be decoupled from CheckDescription class
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51761/ --- Review request for Ambari, Alejandro Fernandez, Di Li, Jayush Luniya, and Nate Cole. Bugs: AMBARI-18050 https://issues.apache.org/jira/browse/AMBARI-18050 Repository: ambari Description --- Currently each pre-req check must add an entry in the CheckDescription enum. This limits the ability for third party stacks and extensions to provide their own pre-req checks. The CheckDescription enum should be rewritten as a class and each pre-req check class should create an instance of it. This will allow stacks and extensions to include their own pre-req checks in separate jar files without requiring changes to ambari-server java code. Diffs - ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java aa8e20c Diff: https://reviews.apache.org/r/51761/diff/ Testing --- mvn clean test -Dtest=AbstractCheckDescriptorTest,UpgradeCheckOrderTest,CheckHelperTest Ran 267 tests in 8.183s Total run:1124 Total errors:0 Total failures:0 Thanks, Tim Thorpe
Re: Review Request 51747: AMBARI-18324: Externalize skip repo url check to ambari.properties instead of hardcoding it in Ambari Java code
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51747/#review148265 --- Ship it! Ship It! - Tim Thorpe On Sept. 8, 2016, 9:11 p.m., Di Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51747/ > --- > > (Updated Sept. 8, 2016, 9:11 p.m.) > > > Review request for Ambari and Tim Thorpe. > > > Bugs: AMBARI-18324 > https://issues.apache.org/jira/browse/AMBARI-18324 > > > Repository: ambari > > > Description > --- > > Externalize skip repo url check to ambari.properties instead of hardcoding it > in Ambari Java code, so that users can define what repo ids to skip repo urls > validation when adding repos for the given stack > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java > 0690ca8 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RepositoryVersionResourceProvider.java > 8d6e6e2 > > ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java > f9b76f8 > > Diff: https://reviews.apache.org/r/51747/diff/ > > > Testing > --- > > existing unit tests > added new unit tests > Patched a trunk cluster with the code change, make sure the properties is not > in the ambari.properties file, verify I can add repos (meaning the default > value kicked in). add the properties with a non-default value, verify I can't > add repos. > > > Thanks, > > Di Li > >
Review Request 51915: AMBARI-18402 - Alert definition should include AGGREGATE source type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51915/ --- Review request for Ambari, Di Li and Jonathan Hurley. Bugs: AMBARI-18402 https://issues.apache.org/jira/browse/AMBARI-18402 Repository: ambari Description --- The current alert definition documentation doesn't include the AGGREGATE source type. Diffs - ambari-server/docs/api/v1/alert-definitions.md fae356a Diff: https://reviews.apache.org/r/51915/diff/ Testing --- None, this is a documentation defect. Thanks, Tim Thorpe
