> On April 5, 2016, 7:53 p.m., Srimanth Gunturi wrote:
> > Tim, thanks for contributing to the stack-advisor feature.
> > 
> > When we started the stack-advisor feature, there was a lot of interconnect 
> > between base service component-layout and configurations (HDFS, YARN, 
> > MapReduce, Hive, HBase) where common calculations were used by multiple 
> > services. To maximise readability and reuse of code, we took the 1 
> > stack-advisor/stack-version approach. This has kept the number of scripts 
> > managable (8 scripts for all versions of HDP stack) to maintain and debug 
> > when problems occur. It is very useful for stacks where a lot of 
> > interconnected services exist and stack-advisor should account for all the 
> > usecases.
> > 
> > Now, coming to this JIRA, it is undeniable that individual service folders 
> > should also be able to contribute to stack-advisor logic - it is useful 
> > when the service is not so dependent on other services and can 
> > independently provide its component-layout and configurations. Also, it is 
> > makes it easier to contribute a rich service by dropping a folder into the 
> > stack.
> > 
> > With that requirement said, I think a smaller change would be to provide a 
> > mechanism by which a service can contribute logic to the stack-version's 
> > stack-advisor. This would allow for both worlds of highly-interconnected 
> > and highly-independent services to coexist in the stack-advisor framework. 
> > For an independent service, it can provide a stack-advisor script which 
> > only dealt with it's component-layout and configuration - that can be 
> > overlaid on top of the stack-version's recommendations/validations.
> > 
> > My vote would be to have such an optional service-level stack-advisor 
> > script, which can be provided only when needed. We can discuss its design 
> > and how best to implement it.

Hi Srimanth, we had a meeting yesterday discussing this and that was the 
consensus.  Service advisors will be able to plug into the stack advisor code.  
It will require some changes to the stacks/stack_advisor.py.

a) For the recommendConfigGroupsConfigurations and recommendConfigurations 
functions, the code for looping through services will need to check for service 
advisors:

      for service in servicesList:
        calculation = self.getServiceConfigurationRecommender(service)
        if calculation is not None:
          calculation(configurations, clusterSummary, services, hosts)
        else:
          advisor = createServiceAdvisor(service)
          if advisor is not None:
            advisor.recommendConfigurations(...)
            
b) For createComponentLayoutRecommendations, the code which processes the 
master and slave services will need to be moved into separate functions: 
getHostsForMasterComponent and getHostsForSlaveComponent.  This will allow 
stack versions to overwrite the details of how those recommendations take place 
without messing up the code which processes service advisors.  The 
createComponentLayoutRecommendations would end up looking something like:

          advisor = createServiceAdvisor(service)
          if advisor is not None:
            advisor.getHostsForMasterComponent(...)
          else
            self.getHostsForMasterComponent(...)
            
c) Change def getComponentLayoutValidations from a no-op to the following:

  def getComponentLayoutValidations(self, services, hosts):

    servicesList = [service["StackServices"]["service_name"] for service in 
services["services"]]
    for service in servicesList:
      advisor = createServiceAdvisor(service)
      if advisor is not None:
        advisor.getComponentLayoutValidations(...)

This also requires the HDP 2.0.6 stack advisor to call 
super.getComponentLayoutValidations

d) Change def getConfigurationsValidationItems from a no-op to the following:

  def getConfigurationsValidationItems(self, services, hosts):

    servicesList = [service["StackServices"]["service_name"] for service in 
services["services"]]
    for service in servicesList:
      advisor = createServiceAdvisor(service)
      if advisor is not None:
        advisor.getConfigurationsValidationItems(...)

This also requires the HDP 2.0.6 stack advisor to call 
super.getConfigurationsValidationItems

I'm sure there are also a number of lesser functions which would need to handle 
things differently.  

One thing I am still struggling with is whether I should create a default 
service advisor class.  The problem with this class is that it would mostly 
duplicate a bunch of functions which are defined in the default stack advisor 
class.  There are basically two options:

1) Create the default service advisor class and live with the duplication (and 
the need to make changes in multiple places)

2) Inherit the custom service advisor classes from the default stack advisor 
and live with the fact that a service is inheriting from the stack.  
(See one of the comments above by Jun Aoki)


- Tim


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review127197
-----------------------------------------------------------


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> -----------------------------------------------------------
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
>     https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py 
> 3a552b8 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/service_advisor_HDP22AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/service_advisor_HDP22HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/service_advisor_HDP22HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/service_advisor_HDP22HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/service_advisor_HDP22KAFKA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/service_advisor_HDP22KNOX.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/service_advisor_HDP22RANGER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/service_advisor_HDP22STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/service_advisor_HDP22TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/service_advisor_HDP22YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/stack_advisor.py
>  bffbc26 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/service_advisor_HDP23HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/service_advisor_HDP23HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/service_advisor_HDP23HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/service_advisor_HDP23KAFKA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/service_advisor_HDP23RANGER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/service_advisor_HDP23RANGER_KMS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/service_advisor_HDP23SPARK.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/service_advisor_HDP23TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/service_advisor_HDP23YARN.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.4/services/stack_advisor.py 
> 985c101 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 9dd5cc7 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 
> af269de 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/AMBARI_METRICS/service_advisor_HDPWIN21AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/FALCON/service_advisor_HDPWIN21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HBASE/service_advisor_HDPWIN21HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/service_advisor_HDPWIN21HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HIVE/service_advisor_HDPWIN21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/OOZIE/service_advisor_HDPWIN21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/STORM/service_advisor_HDPWIN21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/TEZ/service_advisor_HDPWIN21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/YARN/service_advisor_HDPWIN21YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/ZOOKEEPER/service_advisor_HDPWIN21ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/stack_advisor.py 
> b99c484 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/AMBARI_METRICS/service_advisor_HDPWIN22AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HBASE/service_advisor_HDPWIN22HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HDFS/service_advisor_HDPWIN22HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HIVE/service_advisor_HDPWIN22HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/KNOX/service_advisor_HDPWIN22KNOX.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/STORM/service_advisor_HDPWIN22STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/TEZ/service_advisor_HDPWIN22TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/service_advisor_HDPWIN22YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/HBASE/service_advisor_HDPWIN23HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/HDFS/service_advisor_HDPWIN23HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/HIVE/service_advisor_HDPWIN23HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/TEZ/service_advisor_HDPWIN23TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/YARN/service_advisor_HDPWIN23YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/stack_advisor.py 
> 3631c88 
>   ambari-server/src/main/resources/stacks/default_stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/TestStackAdvisor.py 149ae1d 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> e15582e 
>   ambari-server/src/test/python/stacks/2.1/common/test_stack_advisor.py 
> 7ede5a3 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> d95e6c7 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor_perf.py 
> 5b8c30c 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 8932bde 
>   
> ambari-server/src/test/resources/stacks/XYZ/1.0.0/services/YARN/service_advisor_XYZ100YARN.py
>  PRE-CREATION 
>   ambari-server/src/test/resources/stacks/XYZ/1.0.0/services/stack_advisor.py 
> ba140bb 
>   
> ambari-server/src/test/resources/stacks/XYZ/1.0.1/services/YARN/service_advisor_XYZ101YARN.py
>  PRE-CREATION 
>   ambari-server/src/test/resources/stacks/XYZ/1.0.1/services/stack_advisor.py 
> 74a4b31 
>   ambari-server/src/test/resources/stacks/old/HDP206stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP21stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP22stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP23stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP24stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP25stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP26stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/stack_advisor.py PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> -------
> 
> Ran all the non java unit tests.  
> 
> Total run:921
> Total errors:1
> Total failures:0
> 
> The only test which failed also failed on a fresh trunk test run: test_start 
> (test_metrics_grafana.TestMetricsGrafana)
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Reply via email to