Re: Review Request 45338: AMBARI-15053: Stack Featurize YARN and MR services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45338/#review126012
---


Ship it!




Ship It!

- Jayush Luniya


On March 29, 2016, 8:29 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45338/
> ---
> 
> (Updated March 29, 2016, 8:29 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-15053
> https://issues.apache.org/jira/browse/AMBARI-15053
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Featurize HDP specific logic from YARN and MR service
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  f766a82 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  2f0e6bf 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py
>  2966581 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py
>  53b0e53 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  9fc1e32 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
>  fd14d0f 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  1a060de 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py
>  e51ca8a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py
>  83bf460 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  4110d39 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
>  d300279 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  97bd19c 
> 
> Diff: https://reviews.apache.org/r/45338/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test -DskipSurefireTests
> 
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ... SUCCESS [17.789s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.379s]
> [INFO] Ambari Web  SUCCESS [1:28.925s]
> [INFO] Ambari Views .. SUCCESS [7.096s]
> [INFO] Ambari Admin View . SUCCESS [18.829s]
> [INFO] ambari-metrics  SUCCESS [0.358s]
> [INFO] Ambari Metrics Common . SUCCESS [1.222s]
> [INFO] Ambari Metrics Hadoop Sink  SUCCESS [2.142s]
> [INFO] Ambari Metrics Flume Sink . SUCCESS [0.640s]
> [INFO] Ambari Metrics Kafka Sink . SUCCESS [0.800s]
> [INFO] Ambari Metrics Storm Sink . SUCCESS [2.858s]
> [INFO] Ambari Metrics Collector .. SUCCESS [30.150s]
> [INFO] Ambari Metrics Monitor  SUCCESS [3.053s]
> [INFO] Ambari Metrics Grafana  SUCCESS [20.581s]
> [INFO] Ambari Metrics Assembly ... SUCCESS [39.513s]
> [INFO] Ambari Server . SUCCESS [1:42.265s]
> [INFO] Ambari Functional Tests ... SUCCESS [2.830s]
> [INFO] Ambari Agent .. SUCCESS [9.914s]
> [INFO] Ambari Client . SUCCESS [0.286s]
> [INFO] Ambari Python Client .. SUCCESS [1.151s]
> [INFO] Ambari Groovy Client .. SUCCESS [17.175s]
> [INFO] Ambari Shell .. SUCCESS [0.251s]
> [INFO] Ambari Python Shell ... SUCCESS [0.157s]
> [INFO] Ambari Groovy Shell ... SUCCESS [10.952s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 6:20.634s
> [INFO] Finished at: Thu Mar 25 11:22:15 PDT 2016
> [INFO] Final Memory: 112M/531M
> 
> 
> Thanks,
> 
> Juanjo  Marron
> 
>

Re: Review Request 45338: AMBARI-15053: Stack Featurize YARN and MR services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45338/#review125908
---




ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 (line 73)


We are copying Tez, Mapreduce and Slider tarballs under this condition.


- Jayush Luniya


On March 28, 2016, 4:09 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45338/
> ---
> 
> (Updated March 28, 2016, 4:09 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-15053
> https://issues.apache.org/jira/browse/AMBARI-15053
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Featurize HDP specific logic from YARN and MR service
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  f766a82 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  2f0e6bf 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py
>  2966581 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py
>  53b0e53 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  9fc1e32 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
>  fd14d0f 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  52f42b0 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py
>  e51ca8a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py
>  83bf460 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  4110d39 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
>  d300279 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  97bd19c 
> 
> Diff: https://reviews.apache.org/r/45338/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test -DskipSurefireTests
> 
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ... SUCCESS [17.789s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.379s]
> [INFO] Ambari Web  SUCCESS [1:28.925s]
> [INFO] Ambari Views .. SUCCESS [7.096s]
> [INFO] Ambari Admin View . SUCCESS [18.829s]
> [INFO] ambari-metrics  SUCCESS [0.358s]
> [INFO] Ambari Metrics Common . SUCCESS [1.222s]
> [INFO] Ambari Metrics Hadoop Sink  SUCCESS [2.142s]
> [INFO] Ambari Metrics Flume Sink . SUCCESS [0.640s]
> [INFO] Ambari Metrics Kafka Sink . SUCCESS [0.800s]
> [INFO] Ambari Metrics Storm Sink . SUCCESS [2.858s]
> [INFO] Ambari Metrics Collector .. SUCCESS [30.150s]
> [INFO] Ambari Metrics Monitor  SUCCESS [3.053s]
> [INFO] Ambari Metrics Grafana  SUCCESS [20.581s]
> [INFO] Ambari Metrics Assembly ... SUCCESS [39.513s]
> [INFO] Ambari Server . SUCCESS [1:42.265s]
> [INFO] Ambari Functional Tests ... SUCCESS [2.830s]
> [INFO] Ambari Agent .. SUCCESS [9.914s]
> [INFO] Ambari Client . SUCCESS [0.286s]
> [INFO] Ambari Python Client .. SUCCESS [1.151s]
> [INFO] Ambari Groovy Client .. SUCCESS [17.175s]
> [INFO] Ambari Shell .. SUCCESS [0.251s]
> [INFO] Ambari Python Shell ... SUCCESS [0.157s]
> [INFO] Ambari Groovy Shell ... SUCCESS [10.952s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 6:20.634s
> [INFO] Finished at: Thu Mar 25 11:22:15 PDT 2016
> [INFO] Final Memory: 112M/531M
> 
> 
> Thanks,
> 
> Juanjo  Marron
> 
>

Re: Review Request 45338: AMBARI-15053: Stack Featurize YARN and MR services

[Juanjo Marron]

> On March 28, 2016, 6:28 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py,
> >  line 22
> > 
> >
> > Can we remove the import *?
> 
> Juanjo  Marron wrote:
> Most of the service scripts still include the from resource_management 
> import * line. 
> I found 123 matches searching in common-services.
> It would be great to clean up the extra/non use imports, but if 
> @afernandez agrees, I think it is out of the scope of this stack 
> featuraization work

I tried to remove the from resource_management import * line and other specific 
imports are missed, such as Script ot Fail.
It would require additional validation.
Ill create a new JIRA for imports clean up ate common-services level


- Juanjo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45338/#review125695
---


On March 28, 2016, 4:09 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45338/
> ---
> 
> (Updated March 28, 2016, 4:09 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-15053
> https://issues.apache.org/jira/browse/AMBARI-15053
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Featurize HDP specific logic from YARN and MR service
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  f766a82 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  2f0e6bf 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py
>  2966581 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py
>  53b0e53 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  9fc1e32 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
>  fd14d0f 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  52f42b0 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py
>  e51ca8a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py
>  83bf460 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  4110d39 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
>  d300279 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  97bd19c 
> 
> Diff: https://reviews.apache.org/r/45338/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test -DskipSurefireTests
> 
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ... SUCCESS [17.789s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.379s]
> [INFO] Ambari Web  SUCCESS [1:28.925s]
> [INFO] Ambari Views .. SUCCESS [7.096s]
> [INFO] Ambari Admin View . SUCCESS [18.829s]
> [INFO] ambari-metrics  SUCCESS [0.358s]
> [INFO] Ambari Metrics Common . SUCCESS [1.222s]
> [INFO] Ambari Metrics Hadoop Sink  SUCCESS [2.142s]
> [INFO] Ambari Metrics Flume Sink . SUCCESS [0.640s]
> [INFO] Ambari Metrics Kafka Sink . SUCCESS [0.800s]
> [INFO] Ambari Metrics Storm Sink . SUCCESS [2.858s]
> [INFO] Ambari Metrics Collector .. SUCCESS [30.150s]
> [INFO] Ambari Metrics Monitor  SUCCESS [3.053s]
> [INFO] Ambari Metrics Grafana  SUCCESS [20.581s]
> [INFO] Ambari Metrics Assembly ... SUCCESS [39.513s]
> [INFO] Ambari Server . SUCCESS [1:42.265s]
> [INFO] Ambari Functional Tests ... SUCCESS [2.830s]
> [INFO] Ambari Agent .. SUCCESS [9.914s]
> [INFO] Ambari Client . SUCCESS [0.286s]
> [INFO] Ambari Python Client .. SUCCESS [1.151s]
> [INFO] Ambari Groovy Client 

Re: Review Request 45338: AMBARI-15053: Stack Featurize YARN and MR services

[Juanjo Marron]

> On March 28, 2016, 6:28 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py,
> >  line 56
> > 
> >
> > The point of this function was to have a dictionary with the key being 
> > each possible stack name. If we always return "hadoop-client", might as 
> > well just return it as a string no matter what.

The didct value is changing depending the component. The key is always the same 
("HDP" and stack_name now).

Since several services were already pushed to trunk with this same approcah, I 
would propose to solve first stack featurization for all the services and 
create a new JIRA to modify get_stack_to_component() after that.

I think the method def get_stack_to_component(self): in script.py can be 
redefined to obtain the stack_name at script.py level (method def 
get_stack_name()) and to return just a string with the component name at 
service level.


> On March 28, 2016, 6:28 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py,
> >  line 22
> > 
> >
> > Can we remove the import *?

Most of the service scripts still include the from resource_management import * 
line. 
I found 123 matches searching in common-services.
It would be great to clean up the extra/non use imports, but if @afernandez 
agrees, I think it is out of the scope of this stack featuraization work


- Juanjo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45338/#review125695
---


On March 28, 2016, 4:09 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45338/
> ---
> 
> (Updated March 28, 2016, 4:09 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-15053
> https://issues.apache.org/jira/browse/AMBARI-15053
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Featurize HDP specific logic from YARN and MR service
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  f766a82 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  2f0e6bf 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py
>  2966581 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py
>  53b0e53 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  9fc1e32 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
>  fd14d0f 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  52f42b0 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py
>  e51ca8a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py
>  83bf460 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  4110d39 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
>  d300279 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  97bd19c 
> 
> Diff: https://reviews.apache.org/r/45338/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test -DskipSurefireTests
> 
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ... SUCCESS [17.789s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.379s]
> [INFO] Ambari Web  SUCCESS [1:28.925s]
> [INFO] Ambari Views .. SUCCESS [7.096s]
> [INFO] Ambari Admin View . SUCCESS [18.829s]
> [INFO] ambari-metrics  SUCCESS [0.358s]
> [INFO] Ambari Metrics Common . SUCCESS [1.222s]
> [INFO] Ambari Metrics Hadoop Sink  SUCCESS [2.142s]
> [INFO] Ambari Metrics Flume Sink . SUCCESS [0.640s]
> [INFO] Ambari Metrics Kafka Sink . SUCCESS [0.800s]
> [INFO] Ambari Metrics Storm Sink . SUCCESS [2.858s]
> [INFO] Ambari