> On March 2, 2016, 4:56 a.m., Matt wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 1
> > <https://reviews.apache.org/r/44210/diff/1/?file=1275503#file1275503line1>
> >
> >     If I were to add HAWQ 2.0.0's metainfo.xml under HDP 2.3 stack, and say 
> > that it extends from common-services, will this service_advisor be honored?
> 
> Tim Thorpe wrote:
>     I had some difficulty with HAWQ and PXF because they are not actually in 
> the stack and can be added by installing a separate RPM.  In order for HAWQ 
> and PXF to inherit the service_advisor.py files from common-services, they 
> would need to implement mostly empty service advisors under the 
> <stack>/<version>/services/<service> folder that explicitly inherit from the 
> ones in common-services.
> 
> Matt wrote:
>     Is there a way we can avoid having this empty service_advisor?
>     
>     service X in common-services has a service_advisor
>     service X in HDP 2.2 has no service_advisor - should inherit from 
> common-services
>     service X in HDP 2.3 has service_advisor - should inherit from 
> common-services + add new recommendations/validations from HDP 2.3 X's 
> service_advisor
>     
>     Something similar to have themes can be managed or how metainfo.xml has 
> extends property for stack definiton

Hi Matt, I can add the service advisor in the code that reads the 
stack/services from the file system.  This would allow me to know which 
service_advisor.py file should be used (either because it is inherited or 
because it is defined in the current stack version).  This information would 
then need to be passed to the stack advisor in the services.json.  I don't 
think the service advisor should handle implicite inheritance between stack 
versions by that I mean in that your example:

service X in common-services has a service_advisor
service X in HDP 2.2 has no service_advisor - should inherit from 
common-services
service X in HDP 2.3 has service_advisor - only inherits from common-services 
if the service_advisor.py adds that logic 

Basically this means that service X in HDP 2.3 would have 2 options to replace 
the service_advisor.py in common-services or inherit from it and add new 
recommendations/validations.


> On March 2, 2016, 4:56 a.m., Matt wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py,
> >  line 1
> > <https://reviews.apache.org/r/44210/diff/1/?file=1275515#file1275515line1>
> >
> >     If the service_advisor is under HDP 2.0.6's GANGLIA service folder, it 
> > is not required to repeat the stack name, the version and the service on 
> > the filename, for every combination of stack, version and service.
> >     
> >     This would be good enough:
> >     HDP/2.0.6/services/GANGLIA/service_advisor.py
> 
> Tim Thorpe wrote:
>     I tried having the name as you stated but had a problem with python.  For 
> example in 2.3/services/HDFS, I have the following:
>     
>     SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
>     PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/HDFS/')
>     sys.path.append(PARENT_DIR)
>     from service_advisor_HDP22HDFS import *
>     
>     If the file name is the same in 2.3 and 2.2, then "from service_advisor 
> import *" could reference the 2.2 or the 2.3 file.  I tried it with import 
> HDP22HDFSServiceAdvisor as well and both of them failed to find the correct 
> parent class.
>     
>     The other way I could have solved this is to load all the service advisor 
> files from the default_stack_advisor.py walking my way up the stack.  This 
> would have allowed you to avoid the inheritance code in the 
> service_advisor.py files but it would restrict you from inheriting from 
> outside of your stack (unless at that point you started including the 
> inheritance code and the change of file names).

I have figured out a solution for this.  So the naming of the service advisor 
file would always be service_advisor.py without being prefaced with the stack 
name, version and service name.


> On March 2, 2016, 4:56 a.m., Matt wrote:
> > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py, line 
> > 22
> > <https://reviews.apache.org/r/44210/diff/1/?file=1275586#file1275586line22>
> >
> >     If all the stack_advisor.py files are being refactored into service 
> > level advisors, the unit tests should be at service level as well.
> >     
> >     This would bring better modularity for introducing a new service:
> >     - Add a new service to the stack
> >     - Implement the service advisor for that service
> >     - Implement unit test for the service advisor for that service (For the 
> > current patch, the tests are still with stack advisor and not service 
> > advisor)
> 
> Tim Thorpe wrote:
>     I agree but hopefully that can be a future enhancement?  I'd rather get 
> this included and then work on refactoring the test code.

The stack_advisor.py is no longer going to be refactored into service level 
advisors.  The change will only allow for custom services (or later on for core 
services) to be written with service advisors that plug into the stack advisor 
framework.  If and when core services do get refactored into service level 
advisors, the unit tests for those services can be refactored at that time.


- Tim


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review121576
-----------------------------------------------------------


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> -----------------------------------------------------------
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
>     https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py 
> 3a552b8 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/service_advisor_HDP22AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/service_advisor_HDP22HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/service_advisor_HDP22HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/service_advisor_HDP22HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/service_advisor_HDP22KAFKA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/service_advisor_HDP22KNOX.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/service_advisor_HDP22RANGER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/service_advisor_HDP22STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/service_advisor_HDP22TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/service_advisor_HDP22YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/stack_advisor.py
>  bffbc26 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/service_advisor_HDP23HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/service_advisor_HDP23HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/service_advisor_HDP23HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/service_advisor_HDP23KAFKA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/service_advisor_HDP23RANGER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/service_advisor_HDP23RANGER_KMS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/service_advisor_HDP23SPARK.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/service_advisor_HDP23TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/service_advisor_HDP23YARN.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.4/services/stack_advisor.py 
> 985c101 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 9dd5cc7 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 
> af269de 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/AMBARI_METRICS/service_advisor_HDPWIN21AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/FALCON/service_advisor_HDPWIN21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HBASE/service_advisor_HDPWIN21HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/service_advisor_HDPWIN21HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HIVE/service_advisor_HDPWIN21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/OOZIE/service_advisor_HDPWIN21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/STORM/service_advisor_HDPWIN21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/TEZ/service_advisor_HDPWIN21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/YARN/service_advisor_HDPWIN21YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/ZOOKEEPER/service_advisor_HDPWIN21ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/stack_advisor.py 
> b99c484 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/AMBARI_METRICS/service_advisor_HDPWIN22AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HBASE/service_advisor_HDPWIN22HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HDFS/service_advisor_HDPWIN22HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HIVE/service_advisor_HDPWIN22HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/KNOX/service_advisor_HDPWIN22KNOX.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/STORM/service_advisor_HDPWIN22STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/TEZ/service_advisor_HDPWIN22TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/service_advisor_HDPWIN22YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/HBASE/service_advisor_HDPWIN23HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/HDFS/service_advisor_HDPWIN23HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/HIVE/service_advisor_HDPWIN23HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/TEZ/service_advisor_HDPWIN23TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/YARN/service_advisor_HDPWIN23YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.3/services/stack_advisor.py 
> 3631c88 
>   ambari-server/src/main/resources/stacks/default_stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/TestStackAdvisor.py 149ae1d 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> e15582e 
>   ambari-server/src/test/python/stacks/2.1/common/test_stack_advisor.py 
> 7ede5a3 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> d95e6c7 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor_perf.py 
> 5b8c30c 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 8932bde 
>   
> ambari-server/src/test/resources/stacks/XYZ/1.0.0/services/YARN/service_advisor_XYZ100YARN.py
>  PRE-CREATION 
>   ambari-server/src/test/resources/stacks/XYZ/1.0.0/services/stack_advisor.py 
> ba140bb 
>   
> ambari-server/src/test/resources/stacks/XYZ/1.0.1/services/YARN/service_advisor_XYZ101YARN.py
>  PRE-CREATION 
>   ambari-server/src/test/resources/stacks/XYZ/1.0.1/services/stack_advisor.py 
> 74a4b31 
>   ambari-server/src/test/resources/stacks/old/HDP206stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP21stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP22stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP23stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP24stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP25stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/HDP26stack_advisor.py 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/old/stack_advisor.py PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> -------
> 
> Ran all the non java unit tests.  
> 
> Total run:921
> Total errors:1
> Total failures:0
> 
> The only test which failed also failed on a fresh trunk test run: test_start 
> (test_metrics_grafana.TestMetricsGrafana)
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Reply via email to