date:20160527

Re: Review Request 47984: Make storm.topology.submission.notifier.plugin.class property optional

2016-05-27 Thread Sivaguru Chendamaraikannan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47984/
---

(Updated May 28, 2016, 5:15 a.m.)


Review request for Ambari, Srimanth Gunturi and Tom Beerbower.


Bugs: AMBARI-16915
https://issues.apache.org/jira/browse/AMBARI-16915


Repository: ambari


Description
---

The commit, 
https://github.com/apache/ambari/commit/89fd30b40f3108bfcbcc73eb2d74c94a2ba14a7a
 which accidentally made the storm.topology.submission.notifier.plugin property 
mandatory. If the storm defaults yaml config does not set the property a " " 
value is set for the property. This causes storm startup to fail with a Class 
Not Found exception when starting with Ambari. This change fixes the stack 
advisor to not set the property when no value is specified for it through 
command line or storm defaults.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
79314f5 

Diff: https://reviews.apache.org/r/47984/diff/


Testing (updated)
---

mvn test


Thanks,

Sivaguru Chendamaraikannan

Re: Review Request 47996: AMBARI-16942. Take into account reading 'hive.tez.container.size', 'yarn.scheduler.minimum-allocation-mb', 'yarn.nodemanager.resource.memory-mb' & 'tez.am.resource.memory.mb'

2016-05-27 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47996/#review135365
---


Ship it!




Ship It!

- Sumit Mohanty


On May 27, 2016, 11:43 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47996/
> ---
> 
> (Updated May 27, 2016, 11:43 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Daniel Gergely, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-16942
> https://issues.apache.org/jira/browse/AMBARI-16942
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - While doing Hive Server interactive's LLAP related calculations, take into 
> account reading 'hive-site/hive.tez.container.size', 
> 'yarn-site/yarn.scheduler.minimum-allocation-mb', 
> 'yarn-site/yarn.nodemanager.resource.memory-mb' and 
> 'tez-interactive-site/tez.am.resource.memory.mb' if they are changed in 
> current Stack Advisor invocation. Otherwise read from services array.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 837a446 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 3f5da61 
> 
> Diff: https://reviews.apache.org/r/47996/diff/
> 
> 
> Testing
> ---
> 
> - Fixed Python UT.
> - 
> - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml

2016-05-27 Thread Jeff Zhang



> On May 27, 2016, 5:48 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py,
> >  line 63
> > 
> >
> > Why is the tarball source coming from /tmp?

We creat tarball in ambari at installtion instead of installing from rpm. 
Because we want to have full control of what do be included in tarball rather 
than depdends on rpm. please refer 
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47941/#review135273
---


On May 27, 2016, 3:33 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47941/
> ---
> 
> (Updated May 27, 2016, 3:33 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-16920
> https://issues.apache.org/jira/browse/AMBARI-16920
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This a followup ticket for add spark2 stack definition. There's serveral 
> issues:
> 1.  Spark2 thrift server can not started due to miss of 
> spark-thrift-fairscheduler.xml
> 2.  Miss of add spark2 cache file in copy_barball.py
> 3.  Miss the role_commnad_order of spark2
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  286df8d 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
>  ded9959 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py
>  2eae3e7 
>   ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 
> 4a7c1d2 
> 
> Diff: https://reviews.apache.org/r/47941/diff/
> 
> 
> Testing
> ---
> 
> Manually verified.
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 47933: Add conditional constraints for Kerberos identities to control when they are created

2016-05-27 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47933/
---

(Updated May 27, 2016, 7:55 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Oliver Szabo, Sandor 
Magyari, Sumit Mohanty, and Swapan Shridhar.


Changes
---

Fixed merge issue.


Bugs: AMBARI-16437
https://issues.apache.org/jira/browse/AMBARI-16437


Repository: ambari


Description
---

Add conditional constraints for Kerberos identities to control when they are 
created. For example if Kerberos Identity should only be created (and 
distributed) for a component when some other component or service is installed. 

An example of this would be
```
{
  "name": "/HIVE/HIVE_SERVER/hive_server_hive",
  "principal": {
"configuration": "hive-interactive-site/hive.llap.daemon.service.principal"
  },
  "keytab": {
"configuration": "hive-interactive-site/hive.llap.daemon.keytab.file"
  },
  "when" : {
  "contains" : ["services", "HIVE"]
  }
}
```

Note the "`when`" clause. This indicates that this identity should only be 
processed when the set of services contains "HIVE".  An alternative to this 
would be to test the set of components for a certain component.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/collections/Predicate.java 
PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/PredicateUtils.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/AndPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContainsPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContextTransformer.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedMultiplePredicateContainer.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedSinglePredicateContainer.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/EqualsPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/NotPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OperationPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OrPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/PredicateClassFactory.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 c67c55d 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
 0dbd357 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
 bb2ed1c 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
 d31dd21 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 9a58b8d 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/PredicateUtilsTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/AndPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContainsPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContextTransformerTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/EqualsPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/NotPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/OrPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 5393fd6 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
 d80d7cc 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
 0ea7b26 

Diff: https://reviews.apache.org/r/47933/diff/


Testing
---

Manually tested 

# Local test results: PENDING

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 47996: AMBARI-16942. Take into account reading 'hive.tez.container.size', 'yarn.scheduler.minimum-allocation-mb', 'yarn.nodemanager.resource.memory-mb' & 'tez.am.resource.memory.mb'

2016-05-27 Thread Swapan Shridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47996/
---

(Updated May 27, 2016, 11:43 p.m.)


Review request for Ambari, Alejandro Fernandez, Daniel Gergely, and Sumit 
Mohanty.


Summary (updated)
-

AMBARI-16942. Take into account reading 'hive.tez.container.size', 
'yarn.scheduler.minimum-allocation-mb', 'yarn.nodemanager.resource.memory-mb' & 
'tez.am.resource.memory.mb' configs from configurations if they are changed in 
current Stack Advisor invocation.


Bugs: AMBARI-16942
https://issues.apache.org/jira/browse/AMBARI-16942


Repository: ambari


Description
---

- While doing Hive Server interactive's LLAP related calculations, take into 
account reading 'hive-site/hive.tez.container.size', 
'yarn-site/yarn.scheduler.minimum-allocation-mb', 
'yarn-site/yarn.nodemanager.resource.memory-mb' and 
'tez-interactive-site/tez.am.resource.memory.mb' if they are changed in current 
Stack Advisor invocation. Otherwise read from services array.


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
837a446 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 3f5da61 

Diff: https://reviews.apache.org/r/47996/diff/


Testing
---

- Fixed Python UT.
- 
- Python UT passes.


Thanks,

Swapan Shridhar

Review Request 47996: AMBARI-16921. Take into account reading 'hive.tez.container.size', 'yarn.scheduler.minimum-allocation-mb', 'yarn.nodemanager.resource.memory-mb' & 'tez.am.resource.memory.mb' con

2016-05-27 Thread Swapan Shridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47996/
---

Review request for Ambari, Alejandro Fernandez, Daniel Gergely, and Sumit 
Mohanty.


Bugs: AMBARI-16921
https://issues.apache.org/jira/browse/AMBARI-16921


Repository: ambari


Description
---

- While doing Hive Server interactive's LLAP related calculations, take into 
account reading 'hive-site/hive.tez.container.size', 
'yarn-site/yarn.scheduler.minimum-allocation-mb', 
'yarn-site/yarn.nodemanager.resource.memory-mb' and 
'tez-interactive-site/tez.am.resource.memory.mb' if they are changed in current 
Stack Advisor invocation. Otherwise read from services array.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
837a446 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 3f5da61 

Diff: https://reviews.apache.org/r/47996/diff/


Testing
---

- Fixed Python UT.
- 
- Python UT passes.


Thanks,

Swapan Shridhar

Re: Review Request 47993: VDF: support for selecting enabled + default Stacks

2016-05-27 Thread Zhe (Joe) Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47993/#review135348
---


Ship it!




Ship It!

- Zhe (Joe) Wang


On May 27, 2016, 11:14 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47993/
> ---
> 
> (Updated May 27, 2016, 11:14 p.m.)
> 
> 
> Review request for Ambari, Srimanth Gunturi, Xi Wang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-16940
> https://issues.apache.org/jira/browse/AMBARI-16940
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The work revamps the UX for leveraging VDF(Version Definition File) feature 
> of ambari.
> 
> 
> Diffs
> -
> 
>   ambari-admin/src/main/resources/ui/admin-web/app/index.html 9344f15 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/stackVersions/StackVersionsCreateCtrl.js
>  47b72e6 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/stackVersions/StackVersionsEditCtrl.js
>  22ec5ae 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/stackVersions/StackVersionsListCtrl.js
>  3a8233a 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/i18n.config.js 
> 2d51ece 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/AddVersionModal.js
>  PRE-CREATION 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/ConfirmationModal.js
>  4576a40 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Stack.js 
> 96f9c9f 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Utility.js 
> PRE-CREATION 
>   ambari-admin/src/main/resources/ui/admin-web/app/styles/main.css 4434908 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/views/modals/AddVersionModal.html
>  PRE-CREATION 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/views/modals/ConfirmationModal.html
>  b11f6bb 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/views/modals/publicRepoDisabled.html
>  PRE-CREATION 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/views/stackVersions/list.html
>  3de92c1 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/views/stackVersions/stackVersionPage.html
>  f3f2d86 
>   ambari-web/app/assets/test/tests.js c654dac 
>   ambari-web/app/controllers/installer.js 43a5e4a 
>   ambari-web/app/controllers/wizard/step0_controller.js aa3541a 
>   ambari-web/app/controllers/wizard/step1_controller.js 3eb5d1b 
>   ambari-web/app/controllers/wizard/step3_controller.js 38bb5d7 
>   ambari-web/app/controllers/wizard/step4_controller.js 0575f07 
>   ambari-web/app/controllers/wizard/step7_controller.js c3b98b5 
>   ambari-web/app/mappers/stack_mapper.js ab1d6a8 
>   ambari-web/app/messages.js 29ca44c 
>   ambari-web/app/models/repository.js 8ad20ce 
>   ambari-web/app/models/stack.js 6023566 
>   ambari-web/app/models/stack_version/service_simple.js 13ca381 
>   ambari-web/app/routes/installer.js b8b9532 
>   ambari-web/app/styles/application.less 32c88fd 
>   ambari-web/app/styles/common.less 326d404 
>   ambari-web/app/templates/wizard/step1.hbs 56c62a2 
>   
> ambari-web/app/templates/wizard/step1/public_option_disabled_window_body.hbs 
> PRE-CREATION 
>   ambari-web/app/templates/wizard/step1/vdf_upload.hbs PRE-CREATION 
>   ambari-web/app/utils/array_utils.js 643ed67 
>   ambari-web/app/views/wizard/step1_view.js a7af383 
>   ambari-web/karma.conf.js 1a03381 
>   ambari-web/test/controllers/installer_test.js 303c8a4 
>   ambari-web/test/controllers/wizard/step1_test.js PRE-CREATION 
>   ambari-web/test/mixins/common/configs/config_recommendation_parser_test.js 
> 8572772 
>   ambari-web/test/utils/array_utils_test.js ec7c5d8 
>   ambari-web/test/views/common/host_progress_popup_body_view_test.js 282d31c 
>   ambari-web/test/views/wizard/step1_view_test.js 070cb59 
> 
> Diff: https://reviews.apache.org/r/47993/diff/
> 
> 
> Testing
> ---
> 
> Tested ambari-web and ambari-admin view manually:
> Verified that unit tests for ambari-web and ambari-admin passes successfully
> 
> ambari-web:
>   27900 tests complete (29 seconds)
>   154 tests pending
>   
> ambari-admin:
>   phantomJS 1.9.7 (Mac OS X): Executed 71 of 71 SUCCESS (0.467 secs / 0.454 
> secs)
> 
> 
> Thanks,
> 
> Jaimin Jetly
> 
>

Review Request 47993: VDF: support for selecting enabled + default Stacks

2016-05-27 Thread Jaimin Jetly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47993/
---

Review request for Ambari, Srimanth Gunturi, Xi Wang, and Yusaku Sako.


Bugs: AMBARI-16940
https://issues.apache.org/jira/browse/AMBARI-16940


Repository: ambari


Description
---

The work revamps the UX for leveraging VDF(Version Definition File) feature of 
ambari.


Diffs
-

  ambari-admin/src/main/resources/ui/admin-web/app/index.html 9344f15 
  
ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/stackVersions/StackVersionsCreateCtrl.js
 47b72e6 
  
ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/stackVersions/StackVersionsEditCtrl.js
 22ec5ae 
  
ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/stackVersions/StackVersionsListCtrl.js
 3a8233a 
  ambari-admin/src/main/resources/ui/admin-web/app/scripts/i18n.config.js 
2d51ece 
  
ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/AddVersionModal.js
 PRE-CREATION 
  
ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/ConfirmationModal.js
 4576a40 
  ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Stack.js 
96f9c9f 
  ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Utility.js 
PRE-CREATION 
  ambari-admin/src/main/resources/ui/admin-web/app/styles/main.css 4434908 
  
ambari-admin/src/main/resources/ui/admin-web/app/views/modals/AddVersionModal.html
 PRE-CREATION 
  
ambari-admin/src/main/resources/ui/admin-web/app/views/modals/ConfirmationModal.html
 b11f6bb 
  
ambari-admin/src/main/resources/ui/admin-web/app/views/modals/publicRepoDisabled.html
 PRE-CREATION 
  
ambari-admin/src/main/resources/ui/admin-web/app/views/stackVersions/list.html 
3de92c1 
  
ambari-admin/src/main/resources/ui/admin-web/app/views/stackVersions/stackVersionPage.html
 f3f2d86 
  ambari-web/app/assets/test/tests.js c654dac 
  ambari-web/app/controllers/installer.js 43a5e4a 
  ambari-web/app/controllers/wizard/step0_controller.js aa3541a 
  ambari-web/app/controllers/wizard/step1_controller.js 3eb5d1b 
  ambari-web/app/controllers/wizard/step3_controller.js 38bb5d7 
  ambari-web/app/controllers/wizard/step4_controller.js 0575f07 
  ambari-web/app/controllers/wizard/step7_controller.js c3b98b5 
  ambari-web/app/mappers/stack_mapper.js ab1d6a8 
  ambari-web/app/messages.js 29ca44c 
  ambari-web/app/models/repository.js 8ad20ce 
  ambari-web/app/models/stack.js 6023566 
  ambari-web/app/models/stack_version/service_simple.js 13ca381 
  ambari-web/app/routes/installer.js b8b9532 
  ambari-web/app/styles/application.less 32c88fd 
  ambari-web/app/styles/common.less 326d404 
  ambari-web/app/templates/wizard/step1.hbs 56c62a2 
  ambari-web/app/templates/wizard/step1/public_option_disabled_window_body.hbs 
PRE-CREATION 
  ambari-web/app/templates/wizard/step1/vdf_upload.hbs PRE-CREATION 
  ambari-web/app/utils/array_utils.js 643ed67 
  ambari-web/app/views/wizard/step1_view.js a7af383 
  ambari-web/karma.conf.js 1a03381 
  ambari-web/test/controllers/installer_test.js 303c8a4 
  ambari-web/test/controllers/wizard/step1_test.js PRE-CREATION 
  ambari-web/test/mixins/common/configs/config_recommendation_parser_test.js 
8572772 
  ambari-web/test/utils/array_utils_test.js ec7c5d8 
  ambari-web/test/views/common/host_progress_popup_body_view_test.js 282d31c 
  ambari-web/test/views/wizard/step1_view_test.js 070cb59 

Diff: https://reviews.apache.org/r/47993/diff/


Testing
---

Tested ambari-web and ambari-admin view manually:
Verified that unit tests for ambari-web and ambari-admin passes successfully

ambari-web:
  27900 tests complete (29 seconds)
  154 tests pending
  
ambari-admin:
  phantomJS 1.9.7 (Mac OS X): Executed 71 of 71 SUCCESS (0.467 secs / 0.454 
secs)


Thanks,

Jaimin Jetly

Re: Review Request 47428: Changes to Phoenix QueryServer Kerberos configuration

2016-05-27 Thread Josh Elser


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47428/
---

(Updated May 27, 2016, 10:48 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, and 
Srimanth Gunturi.


Changes
---

Just realized that I had one of the UpgradeCatalog240Test cases broken.


Bugs: AMBARI-16171
https://issues.apache.org/jira/browse/AMBARI-16171


Repository: ambari


Description
---

The up-coming version of Phoenix will contain some new functionality to support 
Kerberos authentication of clients via SPNEGO with the Phoenix Query Server 
(PQS).

Presently, Ambari will configure PQS to use the hbase service keytab which will 
result in the SPNEGO authentication failing as the RFC requires that the 
"primary" component of the Kerberos principal for the server is "HTTP". Thus, 
we need to ensure that we switch PQS over to use the spnego.service.keytab as 
the keytab and "HTTP/_HOST@REALM" as the principal.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
 2e857ed 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 0deba5d 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/kerberos.json 
c9536f8 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
6e506a0 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
8c5351f 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
 4dedc98 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 0066e1d 

Diff: https://reviews.apache.org/r/47428/diff/


Testing
---

Unit testing, verified installation with trunk and proper kerberization.


Thanks,

Josh Elser

Re: Review Request 47428: Changes to Phoenix QueryServer Kerberos configuration

2016-05-27 Thread Srimanth Gunturi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47428/#review135341
---


Ship it!




Ship It!

- Srimanth Gunturi


On May 27, 2016, 10:42 p.m., Josh Elser wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47428/
> ---
> 
> (Updated May 27, 2016, 10:42 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, and 
> Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16171
> https://issues.apache.org/jira/browse/AMBARI-16171
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The up-coming version of Phoenix will contain some new functionality to 
> support Kerberos authentication of clients via SPNEGO with the Phoenix Query 
> Server (PQS).
> 
> Presently, Ambari will configure PQS to use the hbase service keytab which 
> will result in the SPNEGO authentication failing as the RFC requires that the 
> "primary" component of the Kerberos principal for the server is "HTTP". Thus, 
> we need to ensure that we switch PQS over to use the spnego.service.keytab as 
> the keytab and "HTTP/_HOST@REALM" as the principal.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
>  2e857ed 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  0deba5d 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/kerberos.json
>  c9536f8 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 6e506a0 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  4dedc98 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 0066e1d 
> 
> Diff: https://reviews.apache.org/r/47428/diff/
> 
> 
> Testing
> ---
> 
> Unit testing, verified installation with trunk and proper kerberization.
> 
> 
> Thanks,
> 
> Josh Elser
> 
>

Re: Review Request 47847: AMBARI-16755 Add spark.driver.extraLibraryPath

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47847/#review135330
---



Pushed to trunk, commit 048ef3fa61be82fdae795c6c466d60c053b1f8f2
branch-2.4, commit c0e8e43f095079784f6b72b2e3e6b926d806eb9c

- Alejandro Fernandez


On May 27, 2016, 9:11 p.m., Weiqing Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47847/
> ---
> 
> (Updated May 27, 2016, 9:11 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16755
> https://issues.apache.org/jira/browse/AMBARI-16755
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add spark.driver.extraLibraryPath in spark-defaults.conf and 
> spark-thrift-sparkconf.conf.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  b507d5e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  c5f3eb6 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  b5742ea 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-defaults.xml
>  5d6c781 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-thrift-sparkconf.xml
>  ce1d159 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
>  ded9959 
> 
> Diff: https://reviews.apache.org/r/47847/diff/
> 
> 
> Testing
> ---
> 
> Tests passed. WARNing messages about unable to load native-hadoop library are 
> gone when users start sparkshell and submit application on Spark.
> 
> 
> Thanks,
> 
> Weiqing Yang
> 
>

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

(Updated May 27, 2016, 5:45 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 617553b

Re: Review Request 47847: AMBARI-16755 Add spark.driver.extraLibraryPath

2016-05-27 Thread Alejandro Fernandez



> On May 27, 2016, 9:36 p.m., Alejandro Fernandez wrote:
> > Ship It!

Let me know when you want me to commit this. Also, is this for Ambari 2.4.0 to 
ship with HDP 2.5?


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47847/#review135327
---


On May 27, 2016, 9:11 p.m., Weiqing Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47847/
> ---
> 
> (Updated May 27, 2016, 9:11 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16755
> https://issues.apache.org/jira/browse/AMBARI-16755
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add spark.driver.extraLibraryPath in spark-defaults.conf and 
> spark-thrift-sparkconf.conf.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  b507d5e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  c5f3eb6 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  b5742ea 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-defaults.xml
>  5d6c781 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-thrift-sparkconf.xml
>  ce1d159 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
>  ded9959 
> 
> Diff: https://reviews.apache.org/r/47847/diff/
> 
> 
> Testing
> ---
> 
> Tests passed. WARNing messages about unable to load native-hadoop library are 
> gone when users start sparkshell and submit application on Spark.
> 
> 
> Thanks,
> 
> Weiqing Yang
> 
>

Re: Review Request 47847: AMBARI-16755 Add spark.driver.extraLibraryPath

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47847/#review135327
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 27, 2016, 9:11 p.m., Weiqing Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47847/
> ---
> 
> (Updated May 27, 2016, 9:11 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16755
> https://issues.apache.org/jira/browse/AMBARI-16755
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add spark.driver.extraLibraryPath in spark-defaults.conf and 
> spark-thrift-sparkconf.conf.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  b507d5e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  c5f3eb6 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  b5742ea 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-defaults.xml
>  5d6c781 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-thrift-sparkconf.xml
>  ce1d159 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
>  ded9959 
> 
> Diff: https://reviews.apache.org/r/47847/diff/
> 
> 
> Testing
> ---
> 
> Tests passed. WARNing messages about unable to load native-hadoop library are 
> gone when users start sparkshell and submit application on Spark.
> 
> 
> Thanks,
> 
> Weiqing Yang
> 
>

Re: Review Request 47428: Changes to Phoenix QueryServer Kerberos configuration

2016-05-27 Thread Josh Elser


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47428/
---

(Updated May 27, 2016, 9:30 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, and 
Srimanth Gunturi.


Changes
---

Srimanth helped me out with some direction on how to update the stackadvisor to 
do the "append" logic instead of an overwrite. I have unit tests added for the 
relevant python changes.

I'm still trying to figure out how to test this on a real installation. It is 
not going well, but I don't want to sit on this patch for any longer.


Bugs: AMBARI-16171
https://issues.apache.org/jira/browse/AMBARI-16171


Repository: ambari


Description
---

The up-coming version of Phoenix will contain some new functionality to support 
Kerberos authentication of clients via SPNEGO with the Phoenix Query Server 
(PQS).

Presently, Ambari will configure PQS to use the hbase service keytab which will 
result in the SPNEGO authentication failing as the RFC requires that the 
"primary" component of the Kerberos principal for the server is "HTTP". Thus, 
we need to ensure that we switch PQS over to use the spnego.service.keytab as 
the keytab and "HTTP/_HOST@REALM" as the principal.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java
 2e857ed 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 0deba5d 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/kerberos.json 
c9536f8 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
6e506a0 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
8c5351f 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
 4dedc98 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 0066e1d 

Diff: https://reviews.apache.org/r/47428/diff/


Testing
---

Unit testing, verified upgrade from 2.2.2


Thanks,

Josh Elser

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

(Updated May 27, 2016, 5:20 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Changes
---

Optimization recommendation from Srimanth


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 617553b

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley

> On May 27, 2016, 4:56 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  line 570
> > 
> >
> > These values feel too small, especially when multiple users are logged 
> > on.

It's a balance, but that's why it's totally configurable. These are just 
processor-based defaults. We don't want too many threads working concurrently, 
otherwise the CPU will be pinned. At the same time, these are now used for 
executing a bunch of very small tasks - the threads are going to blast through 
them pretty fast.

What would you recommend instead of 2 * processors? On my laptop, that's 8. On 
a beefy system with 32 processors, that's 64 threads! 

As I said, these are defaults and we also plan to write a document for 2.4.0 
which outlines all properties and what they are used for and recommended values.

> On May 27, 2016, 4:56 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  line 2507
> > 
> >
> > Should we set a default value of say 1000?

Nope! The idea here is that we do not a bound for the 
ThreadPoolEnabledPropertyProvider anymore - let the queue fill up with small 
tasks and let the core threads handle it. If desired, you can override this 
with the queue size. 

In small systems, there will be so few requests that the queue will never be 
used. In larger systems, we want the queue large enough to hold 10,000's of 
requests.

> On May 27, 2016, 4:56 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/RestMetricsPropertyProvider.java,
> >  line 269
> > 
> >
> > Why was the try { moved down here as opposed to immediately inside the 
> > for loop?

Becuase submitting to the MetricsRetrievalService doesn't throw any exceptions. 
Now, the exceptions are only encountered on processing the parsed result. 
Removed the nested try-block.

> On May 27, 2016, 4:56 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java,
> >  line 792
> > 
> >
> > IMO, this type of test doesn't add much value since anyone that changes 
> > the value will simply go ahead and change the test anyway. Instead, we 
> > should assert that the default values are within a reasonable range.

If a unit test is documented as checking values because they are quite 
important, then changing it at-will is very frowned upon. I would like to keep 
this test, but I can also add tests for "smart" values.

- Jonathan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135316
---

On May 27, 2016, 3:18 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 3:18 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
>

Re: Review Request 47847: AMBARI-16755 Add spark.driver.extraLibraryPath

2016-05-27 Thread Weiqing Yang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47847/
---

(Updated May 27, 2016, 9:11 p.m.)


Review request for Ambari, Sumit Mohanty and Srimanth Gunturi.


Changes
---

1. use a variable to specify paths. 2. make the same change in spark2


Bugs: AMBARI-16755
https://issues.apache.org/jira/browse/AMBARI-16755


Repository: ambari


Description
---

Add spark.driver.extraLibraryPath in spark-defaults.conf and 
spark-thrift-sparkconf.conf.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
 b507d5e 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 c5f3eb6 
  
ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
 b5742ea 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-defaults.xml
 5d6c781 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-thrift-sparkconf.xml
 ce1d159 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
 ded9959 

Diff: https://reviews.apache.org/r/47847/diff/


Testing
---

Tests passed. WARNing messages about unable to load native-hadoop library are 
gone when users start sparkshell and submit application on Spark.


Thanks,

Weiqing Yang

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135316
---




ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (line 570)


These values feel too small, especially when multiple users are logged on.



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (line 2506)


Should we set a default value of say 1000?



ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/RestMetricsPropertyProvider.java
 (line 256)


Why was the try { moved down here as opposed to immediately inside the for 
loop?



ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
 (line 792)


IMO, this type of test doesn't add much value since anyone that changes the 
value will simply go ahead and change the test anyway. Instead, we should 
assert that the default values are within a reasonable range.


- Alejandro Fernandez


On May 27, 2016, 7:18 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 7:18 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the request.
> 
> There are two inherent flaws with this approach:
> 
> - Even with millisecond JMX response times, multiplying this by 100's and 
> then adding parsing overhead causes a noticeable delay in the web client as 
> the federated requests are blocking the main UX request
> 
> - Although there is a threadpool which scales up to service these requests - 
> that only really works for 1 user. With multiple users logged in, you'd need 
> 100's upon 100's

Review Request 47985: Fix for AMBARI-16939, upgrade from 2.2.0 to 2.4.0 on Ubuntu 14.04 / MySQL.

2016-05-27 Thread Balázs Bence Sári


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47985/
---

Review request for Ambari, Jonathan Hurley, Laszlo Puskas, Nate Cole, Oliver 
Szabo, Sandor Magyari, and Sebastian Toader.


Bugs: AMBARI-16939
https://issues.apache.org/jira/browse/AMBARI-16939


Repository: ambari


Description
---

Some 2.2.0 installations carry old (pre Ambari 1.5) FK constraint names in 
their DB schema. Upgrade script knows the constraints under a different name so 
can't drop them. Fix tries to drop the constraint with both their old and new 
names.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 dbbf477 

Diff: https://reviews.apache.org/r/47985/diff/


Testing
---

- Run all unit tests in ambari-server, all passed
- Manually tested upgrading a 2.2.0 cluster having the old constraint names to 
2.4
- Manually tested upgrading a 2.2.0 cluster having the new constraint names to 
2.4


Thanks,

Balázs Bence Sári

Re: Review Request 47858: Cache service advisors when stack advisor is loaded

2016-05-27 Thread Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47858/#review135321
---


Ship it!




Ship It!

- Matt


On May 27, 2016, 12:11 p.m., Lav Jain wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47858/
> ---
> 
> (Updated May 27, 2016, 12:11 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Matt, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-16665
> https://issues.apache.org/jira/browse/AMBARI-16665
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> With the current implementation, service advisors for the same service are 
> loaded multiple times in the stack advisor.
> 
> The service advisors should be cached when the stack advisor is loaded and 
> used where necessary.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 63885d2 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py 
> 1d6a85c 
> 
> Diff: https://reviews.apache.org/r/47858/diff/
> 
> 
> Testing
> ---
> 
> Lavs-MacBook-Pro:common ljain$ python -m discover -v
> test_createComponentLayoutRecommendations_hawq_1_Host 
> (test_stack_advisor.TestHDP23StackAdvisor) ... ServiceAdvisor implementation 
> for service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSTANDBY is recommended on a 3-node cluster ... ServiceAdvisor 
> implementation for service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSEGMENT does not get recommended during Add Service Wizard, 
> when HAWQ has already been installed ... ServiceAdvisor implementation for 
> service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, 
> when HAWQ is selected for installation ... ServiceAdvisor implementation for 
> service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawqsegment_cluster_install 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSEGMENT gets recommended correctly during Cluster Install 
> Wizard, when HAWQ is selected for installation ... ServiceAdvisor 
> implementation for service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_no_hawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test no failures when there are no HAWQ components ... ok
> test_createComponentLayoutRecommendations_pxf_add_service_wizard_already_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that PXF does not get recommended during Add Service Wizard, when PXF 
> has already been installed ... ServiceAdvisor implementation for service PXF 
> was loaded
> ok
> test_createComponentLayoutRecommendations_pxf_add_service_wizard_to_be_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that PXF gets recommended correctly during Add Service Wizard, when PXF 
> is selected for installation ... ServiceAdvisor implementation for service 
> PXF was loaded
> ok
> test_createComponentLayoutRecommendations_pxf_cluster_install 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test that PXF gets recommended correctly during Cluster Install Wizard, when 
> PXF is selected for installation ... ServiceAdvisor implementation for 
> service PXF was loaded
> ok
> test_getComponentLayoutValidations_hawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test layout validations for HAWQ components on a 3-node cluster ... 
> ServiceAdvisor implementation for service HAWQ was loaded
> ok
> test_getComponentLayoutValidations_hawqsegment_not_co_located_with_datanode 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test validation warning for HAWQ segment not colocated with DATANODE ... 
> ServiceAdvisor implementation for service HAWQ was loaded
> ok
> test_getComponentLayoutValidations_nohawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test no failures when there are no HAWQ components on a 3-node cluster ... ok
> test_getComponentLayoutValidations_pxf_co_located_with_nn_and_dn 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test NO warning is generated when PXF is co-located with NAMENODE and 
> DATANODE ... ServiceAdvisor implementation for service PXF was loaded
> ok
> test_getComponentLayoutValidations_pxf_not_co_located_with_dn 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test warning is generated when PXF is not co-located with NAMENODE or 
> DATANODE ... ServiceAdvisor implementation for

Re: Review Request 47975: BP deploy to put default password for hawq_password

2016-05-27 Thread Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47975/#review135320
---


Ship it!




Ship It!

- Matt


On May 27, 2016, 12:48 p.m., bhuvnesh chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47975/
> ---
> 
> (Updated May 27, 2016, 12:48 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, Lav Jain, and Matt.
> 
> 
> Bugs: AMBARI-16936
> https://issues.apache.org/jira/browse/AMBARI-16936
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> BP deploy to put default password for hawq_password
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-env.xml
>  56d2b26 
> 
> Diff: https://reviews.apache.org/r/47975/diff/
> 
> 
> Testing
> ---
> 
> yes.
> 
> 
> Thanks,
> 
> bhuvnesh chaudhary
> 
>

Re: Review Request 47979: Deploy: UI: Hive_metastore not started

2016-05-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47979/#review135319
---


Ship it!




Should add tests covering both hive/hive2.

- Nate Cole


On May 27, 2016, 4:15 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47979/
> ---
> 
> (Updated May 27, 2016, 4:15 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, Nate Cole, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-16938
> https://issues.apache.org/jira/browse/AMBARI-16938
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Deploy ambari cluster using UI with hive for existing mysql setup
> Actual: Hive metastore is failing to start due to unable to connect to DB.
> 2016-05-26 00:20:15,442 - Execute['/usr/jdk64/jdk1.8.0_77/bin/java -cp 
> /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-metastore/lib/mysql-connector-java.jar
>  org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://host/hivedb' 
> hiveuser PROTECTED com.mysql.jdbc.Driver']
> {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 
> 'try_sleep': 10}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  abb8469 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.py
>  5639922 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
>  7554010 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  76a417d 
> 
> Diff: https://reviews.apache.org/r/47979/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley



> On May 27, 2016, 4:06 p.m., Srimanth Gunturi wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/BufferedThreadPoolExecutorCompletionService.java,
> >  line 107
> > 
> >
> > Small optimization: Returning here is going to make us NOT submit any 
> > of the overflown requests till ALL of the already submitted ones are 
> > completed. So if 10 threads + 10-queue got 22 requests, the 2 overflown 
> > requests will not be executed till all 20 got executed. Optimization is to 
> > immediately submit the overflown requests, when we know of completed 
> > requests.

That's a good point - waiting until the super CompletionService is drained to 
process the overflow is not optimal. Let me look into this; we want to be 
careful about submitting to an already exhausted CompletionService since that 
means it'll just come right back into the overflow...


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135301
---


On May 27, 2016, 3:18 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 3:18 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the request.
> 
> There are two inherent flaws with this approach:
> 
> - Even with millisecond JMX response times, multiplying this by 100's and 
> then adding parsing overhead causes a noticeable delay in the web client as 
> the federated requests are blocking the main UX request
> 
> - Although there is a threadpool which scales up to service these requests - 
> that only really works for 1 user. With multiple users logged in, you'd need 
> 100's upon 100's of threads pulling in the same JMX data
> 
> This data should never be queried for directly as part of the

Re: Review Request 47967: Atlas server start failed after Ambari upgrade due to missing solrCloudCli.sh script

2016-05-27 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47967/#review135312
---


Ship it!




Ship It!

- Robert Levas


On May 27, 2016, 1 p.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47967/
> ---
> 
> (Updated May 27, 2016, 1 p.m.)
> 
> 
> Review request for Ambari, John Speidel and Robert Levas.
> 
> 
> Bugs: AMBARI-16932
> https://issues.apache.org/jira/browse/AMBARI-16932
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Steps
> 
> 1.Deploy HDP-2.4.2.0 cluster with Ambari 2.2.2.0 (including Atlas)
> 2.Upgrade Ambari to 2.4.0.0
> 3.Stop and start all services
> 
> Result
> 
> Atlas Metadata server start fails with below error:
>   Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
>  line 165, in 
>   MetadataServer().execute()
>   raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'export 
> JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; 
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z 
> os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
>  --upload-config -d 
> /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs 
> basic_configs -rt 5 -i 10' returned 127. -bash: 
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or 
> directory
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  6f287dc 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  3a9c614 
>   ambari-server/src/test/python/stacks/2.3/configs/default.json ef04248 
>   ambari-server/src/test/python/stacks/2.3/configs/secure.json 166fd6f 
>   ambari-server/src/test/python/stacks/2.5/configs/default.json 0077280 
> 
> Diff: https://reviews.apache.org/r/47967/diff/
> 
> 
> Testing
> ---
> 
> Manual test
> 
> Adjust unit tests
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 47967: Atlas server start failed after Ambari upgrade due to missing solrCloudCli.sh script

2016-05-27 Thread Robert Levas



> On May 27, 2016, 4:08 p.m., Robert Levas wrote:
> > Its not clear how the patch helps to prevent:
> > 
> > ```
> >   Traceback (most recent call last):
> > File 
> > "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
> >  line 165, in 
> >   MetadataServer().execute()
> >   raise Fail(err_msg)
> > resource_management.core.exceptions.Fail: Execution of 'export 
> > JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; 
> > /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z 
> > os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
> >  --upload-config -d 
> > /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf 
> > -cs basic_configs -rt 5 -i 10' returned 127. -bash: 
> > /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or 
> > directory
> > ```
> 
> Tom Beerbower wrote:
> Hey Rob.  Thanks for reviewing!
> 
> Sorry about that.  
> 
> The problem is that after the upgrade the LogSearch script is being 
> called unconditionally and Logsearch is not installed.
> 
> The stack trace in the review description is truncated.  The complete 
> stack is in the Jira, but here is where the script is being called from in 
> metadata.py...
> 
> 
> File 
> "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
>  line 51, in configure
>   metadata()
> File 
> "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py",
>  line 113, in metadata
>   upload_conf_set('basic_configs', random_num)
> File 
> "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py",
>  line 135, in upload_conf_set
>   group=params.user_group)
>   
> Line 113 in metadata.py is where I added the if check to make sure that 
> the configuration was set for Solr and that the LogSearch Solr is actually 
> installed.
> 
> if type == 'server' and params.search_backend_solr and 
> params.has_logsearch_solr:

Hey Tom... thanks for the clarification. I was just making sure something 
wasn't lost.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47967/#review135302
---


On May 27, 2016, 1 p.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47967/
> ---
> 
> (Updated May 27, 2016, 1 p.m.)
> 
> 
> Review request for Ambari, John Speidel and Robert Levas.
> 
> 
> Bugs: AMBARI-16932
> https://issues.apache.org/jira/browse/AMBARI-16932
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Steps
> 
> 1.Deploy HDP-2.4.2.0 cluster with Ambari 2.2.2.0 (including Atlas)
> 2.Upgrade Ambari to 2.4.0.0
> 3.Stop and start all services
> 
> Result
> 
> Atlas Metadata server start fails with below error:
>   Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
>  line 165, in 
>   MetadataServer().execute()
>   raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'export 
> JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; 
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z 
> os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
>  --upload-config -d 
> /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs 
> basic_configs -rt 5 -i 10' returned 127. -bash: 
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or 
> directory
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  6f287dc 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  3a9c614 
>   ambari-server/src/test/python/stacks/2.3/configs/default.json ef04248 
>   ambari-server/src/test/python/stacks/2.3/configs/secure.json 166fd6f 
>   ambari-server/src/test/python/stacks/2.5/configs/default.json 0077280 
> 
> Diff: https://reviews.apache.org/r/47967/diff/
> 
> 
> Testing
> ---
> 
> Manual test
> 
> Adjust unit tests
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 47976: LDAP sync cannot handle if the member attribute value is not DN or id

2016-05-27 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47976/#review135310
---


Ship it!




Ship It!

- Robert Levas


On May 27, 2016, 4:14 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47976/
> ---
> 
> (Updated May 27, 2016, 4:14 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Robert Levas, Robert Nettleton, 
> and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16875
> https://issues.apache.org/jira/browse/AMBARI-16875
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In some rare cases, member attribute value for a group/user can be 
> constructed. (not baseDN/uid, sometimes ldap proxies does that)
> 
> Added 2 feature to fix these problems (to manipulate queries that are used 
> during sync):
> 
> 2.1.) use regexp to get the useful informations from a custom member 
> attribute value: (for groups/users)
> "authentication.ldap.sync.userMemberReplacePattern"
> "authentication.ldap.sync.groupMemberReplacePattern"
> 
> e.g.:
> member:

Review Request 47984: Make storm.topology.submission.notifier.plugin.class property optional

2016-05-27 Thread Sivaguru Chendamaraikannan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47984/
---

Review request for Ambari, Srimanth Gunturi and Tom Beerbower.


Bugs: AMBARI-16915
https://issues.apache.org/jira/browse/AMBARI-16915


Repository: ambari


Description
---

The commit, 
https://github.com/apache/ambari/commit/89fd30b40f3108bfcbcc73eb2d74c94a2ba14a7a
 which accidentally made the storm.topology.submission.notifier.plugin property 
mandatory. If the storm defaults yaml config does not set the property a " " 
value is set for the property. This causes storm startup to fail with a Class 
Not Found exception when starting with Ambari. This change fixes the stack 
advisor to not set the property when no value is specified for it through 
command line or storm defaults.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
79314f5 

Diff: https://reviews.apache.org/r/47984/diff/


Testing
---


Thanks,

Sivaguru Chendamaraikannan

Re: Review Request 47967: Atlas server start failed after Ambari upgrade due to missing solrCloudCli.sh script

2016-05-27 Thread Tom Beerbower



> On May 27, 2016, 8:08 p.m., Robert Levas wrote:
> > Its not clear how the patch helps to prevent:
> > 
> > ```
> >   Traceback (most recent call last):
> > File 
> > "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
> >  line 165, in 
> >   MetadataServer().execute()
> >   raise Fail(err_msg)
> > resource_management.core.exceptions.Fail: Execution of 'export 
> > JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; 
> > /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z 
> > os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
> >  --upload-config -d 
> > /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf 
> > -cs basic_configs -rt 5 -i 10' returned 127. -bash: 
> > /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or 
> > directory
> > ```

Hey Rob.  Thanks for reviewing!

Sorry about that.  

The problem is that after the upgrade the LogSearch script is being called 
unconditionally and Logsearch is not installed.

The stack trace in the review description is truncated.  The complete stack is 
in the Jira, but here is where the script is being called from in metadata.py...


File 
"/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
 line 51, in configure
  metadata()
File 
"/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py",
 line 113, in metadata
  upload_conf_set('basic_configs', random_num)
File 
"/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py",
 line 135, in upload_conf_set
  group=params.user_group)
  
Line 113 in metadata.py is where I added the if check to make sure that the 
configuration was set for Solr and that the LogSearch Solr is actually 
installed.

if type == 'server' and params.search_backend_solr and 
params.has_logsearch_solr:


- Tom


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47967/#review135302
---


On May 27, 2016, 5 p.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47967/
> ---
> 
> (Updated May 27, 2016, 5 p.m.)
> 
> 
> Review request for Ambari, John Speidel and Robert Levas.
> 
> 
> Bugs: AMBARI-16932
> https://issues.apache.org/jira/browse/AMBARI-16932
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Steps
> 
> 1.Deploy HDP-2.4.2.0 cluster with Ambari 2.2.2.0 (including Atlas)
> 2.Upgrade Ambari to 2.4.0.0
> 3.Stop and start all services
> 
> Result
> 
> Atlas Metadata server start fails with below error:
>   Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
>  line 165, in 
>   MetadataServer().execute()
>   raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'export 
> JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; 
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z 
> os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
>  --upload-config -d 
> /usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs 
> basic_configs -rt 5 -i 10' returned 127. -bash: 
> /usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or 
> directory
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  6f287dc 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  3a9c614 
>   ambari-server/src/test/python/stacks/2.3/configs/default.json ef04248 
>   ambari-server/src/test/python/stacks/2.3/configs/secure.json 166fd6f 
>   ambari-server/src/test/python/stacks/2.5/configs/default.json 0077280 
> 
> Diff: https://reviews.apache.org/r/47967/diff/
> 
> 
> Testing
> ---
> 
> Manual test
> 
> Adjust unit tests
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 47975: BP deploy to put default password for hawq_password

2016-05-27 Thread Lav Jain


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47975/#review135306
---


Ship it!




Ship It!

- Lav Jain


On May 27, 2016, 7:48 p.m., bhuvnesh chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47975/
> ---
> 
> (Updated May 27, 2016, 7:48 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, Lav Jain, and Matt.
> 
> 
> Bugs: AMBARI-16936
> https://issues.apache.org/jira/browse/AMBARI-16936
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> BP deploy to put default password for hawq_password
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-env.xml
>  56d2b26 
> 
> Diff: https://reviews.apache.org/r/47975/diff/
> 
> 
> Testing
> ---
> 
> yes.
> 
> 
> Thanks,
> 
> bhuvnesh chaudhary
> 
>

Re: Review Request 47979: Deploy: UI: Hive_metastore not started

2016-05-27 Thread Myroslav Papirkovskyy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47979/#review135305
---


Ship it!




Ship It!

- Myroslav Papirkovskyy


On Травень 27, 2016, 11:15 після полудня, Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47979/
> ---
> 
> (Updated Травень 27, 2016, 11:15 після полудня)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, Nate Cole, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-16938
> https://issues.apache.org/jira/browse/AMBARI-16938
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Deploy ambari cluster using UI with hive for existing mysql setup
> Actual: Hive metastore is failing to start due to unable to connect to DB.
> 2016-05-26 00:20:15,442 - Execute['/usr/jdk64/jdk1.8.0_77/bin/java -cp 
> /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-metastore/lib/mysql-connector-java.jar
>  org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://host/hivedb' 
> hiveuser PROTECTED com.mysql.jdbc.Driver']
> {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 
> 'try_sleep': 10}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  abb8469 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.py
>  5639922 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
>  7554010 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  76a417d 
> 
> Diff: https://reviews.apache.org/r/47979/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Review Request 47979: Deploy: UI: Hive_metastore not started

2016-05-27 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47979/
---

Review request for Ambari, Andrew Onischuk, Dmytro Sen, Nate Cole, and Sumit 
Mohanty.


Bugs: AMBARI-16938
https://issues.apache.org/jira/browse/AMBARI-16938


Repository: ambari


Description
---

Deploy ambari cluster using UI with hive for existing mysql setup
Actual: Hive metastore is failing to start due to unable to connect to DB.
2016-05-26 00:20:15,442 - Execute['/usr/jdk64/jdk1.8.0_77/bin/java -cp 
/usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-metastore/lib/mysql-connector-java.jar
 org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://host/hivedb' 
hiveuser PROTECTED com.mysql.jdbc.Driver']
{'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 
'try_sleep': 10}


Diffs
-

  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
 abb8469 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_interactive.py
 5639922 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
 7554010 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
 76a417d 

Diff: https://reviews.apache.org/r/47979/diff/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 47933: Add conditional constraints for Kerberos identities to control when they are created

2016-05-27 Thread Oliver Szabo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47933/#review135303
---


Ship it!




Ship It!

- Oliver Szabo


On May 27, 2016, 7:58 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47933/
> ---
> 
> (Updated May 27, 2016, 7:58 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Oliver Szabo, Sandor 
> Magyari, Sumit Mohanty, and Swapan Shridhar.
> 
> 
> Bugs: AMBARI-16437
> https://issues.apache.org/jira/browse/AMBARI-16437
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add conditional constraints for Kerberos identities to control when they are 
> created. For example if Kerberos Identity should only be created (and 
> distributed) for a component when some other component or service is 
> installed. 
> 
> An example of this would be
> ```
> {
>   "name": "/HIVE/HIVE_SERVER/hive_server_hive",
>   "principal": {
> "configuration": 
> "hive-interactive-site/hive.llap.daemon.service.principal"
>   },
>   "keytab": {
> "configuration": "hive-interactive-site/hive.llap.daemon.keytab.file"
>   },
>   "when" : {
>   "contains" : ["services", "HIVE"]
>   }
> }
> ```
> 
> Note the "`when`" clause. This indicates that this identity should only be 
> processed when the set of services contains "HIVE".  An alternative to this 
> would be to test the set of components for a certain component.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/Predicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/PredicateUtils.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/AndPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContainsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContextTransformer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedMultiplePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedSinglePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/EqualsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/NotPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OperationPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OrPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/PredicateClassFactory.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  c67c55d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  0dbd357 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
>  bb2ed1c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
>  d31dd21 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/PredicateUtilsTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/AndPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContainsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContextTransformerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/EqualsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/NotPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/OrPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  5393fd6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
>  d80d7cc 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
>  0ea7b26 
> 
> Diff: https://reviews.apache.org/r/47933/diff/
> 
> 
> Testing
> ---
> 
> Manually tested 
> 
> # Local test results: PENDING
> 
> #

Review Request 47976: LDAP sync cannot handle if the member attribute value is not DN or id

2016-05-27 Thread Oliver Szabo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47976/
---

Review request for Ambari, Daniel Gergely, Robert Levas, Robert Nettleton, and 
Sebastian Toader.


Bugs: AMBARI-16875
https://issues.apache.org/jira/browse/AMBARI-16875


Repository: ambari


Description
---

In some rare cases, member attribute value for a group/user can be constructed. 
(not baseDN/uid, sometimes ldap proxies does that)

Added 2 feature to fix these problems (to manipulate queries that are used 
during sync):

2.1.) use regexp to get the useful informations from a custom member attribute 
value: (for groups/users)
"authentication.ldap.sync.userMemberReplacePattern"
"authentication.ldap.sync.groupMemberReplacePattern"

e.g.:
member:

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Srimanth Gunturi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135301
---




ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/BufferedThreadPoolExecutorCompletionService.java
 (line 107)


Small optimization: Returning here is going to make us NOT submit any of 
the overflown requests till ALL of the already submitted ones are completed. So 
if 10 threads + 10-queue got 22 requests, the 2 overflown requests will not be 
executed till all 20 got executed. Optimization is to immediately submit the 
overflown requests, when we know of completed requests.


- Srimanth Gunturi


On May 27, 2016, 7:18 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 7:18 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the request.
> 
> There are two inherent flaws with this approach:
> 
> - Even with millisecond JMX response times, multiplying this by 100's and 
> then adding parsing overhead causes a noticeable delay in the web client as 
> the federated requests are blocking the main UX request
> 
> - Although there is a threadpool which scales up to service these requests - 
> that only really works for 1 user. With multiple users logged in, you'd need 
> 100's upon 100's of threads pulling in the same JMX data
> 
> This data should never be queried for directly as part of the incoming REST 
> requests. Instead, an autonomous pool of threads should be constantly 
> retrieving these point-in-time metrics and updating a cache. The cache is 
> then used to service all live REST requests. 
> - On the first request to a resource, a cache miss occurs and no data is 
> returned. I think this is acceptable since metrics take a few moments to 
> populate anyway right now. As the

Re: Review Request 47933: Add conditional constraints for Kerberos identities to control when they are created

2016-05-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47933/#review135300
---


Ship it!




Ship It!

- Nate Cole


On May 27, 2016, 3:58 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47933/
> ---
> 
> (Updated May 27, 2016, 3:58 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Oliver Szabo, Sandor 
> Magyari, Sumit Mohanty, and Swapan Shridhar.
> 
> 
> Bugs: AMBARI-16437
> https://issues.apache.org/jira/browse/AMBARI-16437
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add conditional constraints for Kerberos identities to control when they are 
> created. For example if Kerberos Identity should only be created (and 
> distributed) for a component when some other component or service is 
> installed. 
> 
> An example of this would be
> ```
> {
>   "name": "/HIVE/HIVE_SERVER/hive_server_hive",
>   "principal": {
> "configuration": 
> "hive-interactive-site/hive.llap.daemon.service.principal"
>   },
>   "keytab": {
> "configuration": "hive-interactive-site/hive.llap.daemon.keytab.file"
>   },
>   "when" : {
>   "contains" : ["services", "HIVE"]
>   }
> }
> ```
> 
> Note the "`when`" clause. This indicates that this identity should only be 
> processed when the set of services contains "HIVE".  An alternative to this 
> would be to test the set of components for a certain component.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/Predicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/PredicateUtils.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/AndPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContainsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContextTransformer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedMultiplePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedSinglePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/EqualsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/NotPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OperationPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OrPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/collections/functors/PredicateClassFactory.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  c67c55d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  0dbd357 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
>  bb2ed1c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
>  d31dd21 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/PredicateUtilsTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/AndPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContainsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContextTransformerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/EqualsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/NotPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/collections/functors/OrPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  5393fd6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
>  d80d7cc 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
>  0ea7b26 
> 
> Diff: https://reviews.apache.org/r/47933/diff/
> 
> 
> Testing
> ---
> 
> Manually tested 
> 
> # Local test results: PENDING
> 
> #

Re: Review Request 47933: Add conditional constraints for Kerberos identities to control when they are created

2016-05-27 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47933/
---

(Updated May 27, 2016, 3:58 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Oliver Szabo, Sandor 
Magyari, Sumit Mohanty, and Swapan Shridhar.


Changes
---

Moved `org.apache.ambari.server.state.kerberos.eval` to 
`org.apache.ambari.server.collections`.


Bugs: AMBARI-16437
https://issues.apache.org/jira/browse/AMBARI-16437


Repository: ambari


Description
---

Add conditional constraints for Kerberos identities to control when they are 
created. For example if Kerberos Identity should only be created (and 
distributed) for a component when some other component or service is installed. 

An example of this would be
```
{
  "name": "/HIVE/HIVE_SERVER/hive_server_hive",
  "principal": {
"configuration": "hive-interactive-site/hive.llap.daemon.service.principal"
  },
  "keytab": {
"configuration": "hive-interactive-site/hive.llap.daemon.keytab.file"
  },
  "when" : {
  "contains" : ["services", "HIVE"]
  }
}
```

Note the "`when`" clause. This indicates that this identity should only be 
processed when the set of services contains "HIVE".  An alternative to this 
would be to test the set of components for a certain component.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/collections/Predicate.java 
PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/PredicateUtils.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/AndPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContainsPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/ContextTransformer.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedMultiplePredicateContainer.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/DelegatedSinglePredicateContainer.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/EqualsPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/NotPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OperationPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/OrPredicate.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/collections/functors/PredicateClassFactory.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 c67c55d 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
 0dbd357 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
 bb2ed1c 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
 d31dd21 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/PredicateUtilsTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/AndPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContainsPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/ContextTransformerTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/EqualsPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/NotPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/collections/functors/OrPredicateTest.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 5393fd6 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
 d80d7cc 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
 0ea7b26 

Diff: https://reviews.apache.org/r/47933/diff/


Testing
---

Manually tested 

# Local test results: PENDING

# Jenkins test results: PENDING


Thanks,

Robert Levas

Review Request 47975: BP deploy to put default password for hawq_password

2016-05-27 Thread bhuvnesh chaudhary


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47975/
---

Review request for Ambari, Alexander Denissov, Lav Jain, and Matt.


Bugs: AMBARI-16936
https://issues.apache.org/jira/browse/AMBARI-16936


Repository: ambari


Description
---

BP deploy to put default password for hawq_password


Diffs
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-env.xml
 56d2b26 

Diff: https://reviews.apache.org/r/47975/diff/


Testing
---

yes.


Thanks,

bhuvnesh chaudhary

Re: Review Request 47858: Cache service advisors when stack advisor is loaded

2016-05-27 Thread Tim Thorpe


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47858/#review135294
---


Ship it!




Ship It!

- Tim Thorpe


On May 27, 2016, 7:11 p.m., Lav Jain wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47858/
> ---
> 
> (Updated May 27, 2016, 7:11 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Matt, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-16665
> https://issues.apache.org/jira/browse/AMBARI-16665
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> With the current implementation, service advisors for the same service are 
> loaded multiple times in the stack advisor.
> 
> The service advisors should be cached when the stack advisor is loaded and 
> used where necessary.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 63885d2 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py 
> 1d6a85c 
> 
> Diff: https://reviews.apache.org/r/47858/diff/
> 
> 
> Testing
> ---
> 
> Lavs-MacBook-Pro:common ljain$ python -m discover -v
> test_createComponentLayoutRecommendations_hawq_1_Host 
> (test_stack_advisor.TestHDP23StackAdvisor) ... ServiceAdvisor implementation 
> for service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSTANDBY is recommended on a 3-node cluster ... ServiceAdvisor 
> implementation for service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSEGMENT does not get recommended during Add Service Wizard, 
> when HAWQ has already been installed ... ServiceAdvisor implementation for 
> service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, 
> when HAWQ is selected for installation ... ServiceAdvisor implementation for 
> service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_hawqsegment_cluster_install 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test that HAWQSEGMENT gets recommended correctly during Cluster Install 
> Wizard, when HAWQ is selected for installation ... ServiceAdvisor 
> implementation for service HAWQ was loaded
> ok
> test_createComponentLayoutRecommendations_no_hawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test no failures when there are no HAWQ components ... ok
> test_createComponentLayoutRecommendations_pxf_add_service_wizard_already_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that PXF does not get recommended during Add Service Wizard, when PXF 
> has already been installed ... ServiceAdvisor implementation for service PXF 
> was loaded
> ok
> test_createComponentLayoutRecommendations_pxf_add_service_wizard_to_be_installed
>  (test_stack_advisor.TestHDP23StackAdvisor)
> Test that PXF gets recommended correctly during Add Service Wizard, when PXF 
> is selected for installation ... ServiceAdvisor implementation for service 
> PXF was loaded
> ok
> test_createComponentLayoutRecommendations_pxf_cluster_install 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test that PXF gets recommended correctly during Cluster Install Wizard, when 
> PXF is selected for installation ... ServiceAdvisor implementation for 
> service PXF was loaded
> ok
> test_getComponentLayoutValidations_hawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test layout validations for HAWQ components on a 3-node cluster ... 
> ServiceAdvisor implementation for service HAWQ was loaded
> ok
> test_getComponentLayoutValidations_hawqsegment_not_co_located_with_datanode 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test validation warning for HAWQ segment not colocated with DATANODE ... 
> ServiceAdvisor implementation for service HAWQ was loaded
> ok
> test_getComponentLayoutValidations_nohawq_3_Hosts 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test no failures when there are no HAWQ components on a 3-node cluster ... ok
> test_getComponentLayoutValidations_pxf_co_located_with_nn_and_dn 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test NO warning is generated when PXF is co-located with NAMENODE and 
> DATANODE ... ServiceAdvisor implementation for service PXF was loaded
> ok
> test_getComponentLayoutValidations_pxf_not_co_located_with_dn 
> (test_stack_advisor.TestHDP23StackAdvisor)
> Test warning is generated when PXF is not co-located with NAMENODE or 
> DATANODE ... ServiceAdvisor implementation for

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135291
---


Ship it!




Ship It!

- Robert Levas


On May 27, 2016, 3:18 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 3:18 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the request.
> 
> There are two inherent flaws with this approach:
> 
> - Even with millisecond JMX response times, multiplying this by 100's and 
> then adding parsing overhead causes a noticeable delay in the web client as 
> the federated requests are blocking the main UX request
> 
> - Although there is a threadpool which scales up to service these requests - 
> that only really works for 1 user. With multiple users logged in, you'd need 
> 100's upon 100's of threads pulling in the same JMX data
> 
> This data should never be queried for directly as part of the incoming REST 
> requests. Instead, an autonomous pool of threads should be constantly 
> retrieving these point-in-time metrics and updating a cache. The cache is 
> then used to service all live REST requests. 
> - On the first request to a resource, a cache miss occurs and no data is 
> returned. I think this is acceptable since metrics take a few moments to 
> populate anyway right now. As the web client polls, the next request should 
> pickup the newly cached metrics.
> - Only URLs which are being asked for by incoming REST requests should be 
> considered for retrieval. After sometime, if they haven't been requested, 
> then the headless threadpool can stop trying to update their data
> - All JMX data will be parsed and stored in-memory, in an expiring cache
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
> 186e272 
>   
>

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

(Updated May 27, 2016, 3:18 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 617553b

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley



> On May 27, 2016, 3:04 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/RestMetricsPropertyProvider.java,
> >  line 60
> > 
> >
> > "returns is as a resource property" -> "returns it as a resource 
> > property"

Fixed.


> On May 27, 2016, 3:04 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/jmx/JMXPropertyProvider.java,
> >  line 315
> > 
> >
> > This method name is confusing... nothing is being returned yet it's 
> > name is `get`.  Maybe it should be renamed `updateMetricResource` or 
> > something like that.

Due to the nature of the backport that's going to be required, I'm trying not 
to change method names.


> On May 27, 2016, 3:04 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractProviderModule.java,
> >  line 1220
> > 
> >
> > Seems like 
> > ```
> > PROPERTY_HDFS_HTTP_POLICY_VALUE_HTTPS_ONLY.equals(value)
> > ```
> > would be more efficient since 
> > `PROPERTY_HDFS_HTTP_POLICY_VALUE_HTTPS_ONLY` should not be null and you 
> > avoid an additional method call.

Seeing as though "value" is a String, I agree. Will do.


> On May 27, 2016, 3:04 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  line 52
> > 
> >
> > Where is 
> > `org.apache.ambari.server.state.services.MetricsRetrievalService`?
> 
> Robert Levas wrote:
> I am not sure how I failed to find the class.

RB's fault... it wasn't showing it unless I did a different kind of patch.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135257
---


On May 27, 2016, 2:17 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 2:17 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
>

Re: Review Request 47858: Cache service advisors when stack advisor is loaded

2016-05-27 Thread Lav Jain


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47858/
---

(Updated May 27, 2016, 7:11 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Matt, and Tim Thorpe.


Bugs: AMBARI-16665
https://issues.apache.org/jira/browse/AMBARI-16665


Repository: ambari


Description
---

With the current implementation, service advisors for the same service are 
loaded multiple times in the stack advisor.

The service advisors should be cached when the stack advisor is loaded and used 
where necessary.


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/stack_advisor.py 63885d2 
  ambari-server/src/test/python/stacks/2.3/HAWQ/test_service_advisor.py 1d6a85c 

Diff: https://reviews.apache.org/r/47858/diff/


Testing
---

Lavs-MacBook-Pro:common ljain$ python -m discover -v
test_createComponentLayoutRecommendations_hawq_1_Host 
(test_stack_advisor.TestHDP23StackAdvisor) ... ServiceAdvisor implementation 
for service HAWQ was loaded
ok
test_createComponentLayoutRecommendations_hawq_3_Hosts 
(test_stack_advisor.TestHDP23StackAdvisor)
Test that HAWQSTANDBY is recommended on a 3-node cluster ... ServiceAdvisor 
implementation for service HAWQ was loaded
ok
test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_already_installed
 (test_stack_advisor.TestHDP23StackAdvisor)
Test that HAWQSEGMENT does not get recommended during Add Service Wizard, when 
HAWQ has already been installed ... ServiceAdvisor implementation for service 
HAWQ was loaded
ok
test_createComponentLayoutRecommendations_hawqsegment_add_service_wizard_to_be_installed
 (test_stack_advisor.TestHDP23StackAdvisor)
Test that HAWQSEGMENT gets recommended correctly during Add Service Wizard, 
when HAWQ is selected for installation ... ServiceAdvisor implementation for 
service HAWQ was loaded
ok
test_createComponentLayoutRecommendations_hawqsegment_cluster_install 
(test_stack_advisor.TestHDP23StackAdvisor)
Test that HAWQSEGMENT gets recommended correctly during Cluster Install Wizard, 
when HAWQ is selected for installation ... ServiceAdvisor implementation for 
service HAWQ was loaded
ok
test_createComponentLayoutRecommendations_no_hawq_3_Hosts 
(test_stack_advisor.TestHDP23StackAdvisor)
Test no failures when there are no HAWQ components ... ok
test_createComponentLayoutRecommendations_pxf_add_service_wizard_already_installed
 (test_stack_advisor.TestHDP23StackAdvisor)
Test that PXF does not get recommended during Add Service Wizard, when PXF has 
already been installed ... ServiceAdvisor implementation for service PXF was 
loaded
ok
test_createComponentLayoutRecommendations_pxf_add_service_wizard_to_be_installed
 (test_stack_advisor.TestHDP23StackAdvisor)
Test that PXF gets recommended correctly during Add Service Wizard, when PXF is 
selected for installation ... ServiceAdvisor implementation for service PXF was 
loaded
ok
test_createComponentLayoutRecommendations_pxf_cluster_install 
(test_stack_advisor.TestHDP23StackAdvisor)
Test that PXF gets recommended correctly during Cluster Install Wizard, when 
PXF is selected for installation ... ServiceAdvisor implementation for service 
PXF was loaded
ok
test_getComponentLayoutValidations_hawq_3_Hosts 
(test_stack_advisor.TestHDP23StackAdvisor)
Test layout validations for HAWQ components on a 3-node cluster ... 
ServiceAdvisor implementation for service HAWQ was loaded
ok
test_getComponentLayoutValidations_hawqsegment_not_co_located_with_datanode 
(test_stack_advisor.TestHDP23StackAdvisor)
Test validation warning for HAWQ segment not colocated with DATANODE ... 
ServiceAdvisor implementation for service HAWQ was loaded
ok
test_getComponentLayoutValidations_nohawq_3_Hosts 
(test_stack_advisor.TestHDP23StackAdvisor)
Test no failures when there are no HAWQ components on a 3-node cluster ... ok
test_getComponentLayoutValidations_pxf_co_located_with_nn_and_dn 
(test_stack_advisor.TestHDP23StackAdvisor)
Test NO warning is generated when PXF is co-located with NAMENODE and DATANODE 
... ServiceAdvisor implementation for service PXF was loaded
ok
test_getComponentLayoutValidations_pxf_not_co_located_with_dn 
(test_stack_advisor.TestHDP23StackAdvisor)
Test warning is generated when PXF is not co-located with NAMENODE or DATANODE 
... ServiceAdvisor implementation for service PXF was loaded
ok
test_getComponentLayoutValidations_pxf_not_co_located_with_nn 
(test_stack_advisor.TestHDP23StackAdvisor)
Test warning is generated when PXF is not co-located with NAMENODE ... 
ServiceAdvisor implementation for service PXF was loaded
ok
test_getComponentLayoutValidations_pxf_not_co_located_with_nn_or_dn 
(test_stack_advisor.TestHDP23StackAdvisor)
Test warning is generated when PXF is not co-located with NAMENODE or DATANODE 
... ServiceAdvisor implementation for service PXF was loaded
ok

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Robert Levas



> On May 27, 2016, 3:04 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  line 52
> > 
> >
> > Where is 
> > `org.apache.ambari.server.state.services.MetricsRetrievalService`?

I am not sure how I failed to find the class.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135257
---


On May 27, 2016, 2:17 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 2:17 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the request.
> 
> There are two inherent flaws with this approach:
> 
> - Even with millisecond JMX response times, multiplying this by 100's and 
> then adding parsing overhead causes a noticeable delay in the web client as 
> the federated requests are blocking the main UX request
> 
> - Although there is a threadpool which scales up to service these requests - 
> that only really works for 1 user. With multiple users logged in, you'd need 
> 100's upon 100's of threads pulling in the same JMX data
> 
> This data should never be queried for directly as part of the incoming REST 
> requests. Instead, an autonomous pool of threads should be constantly 
> retrieving these point-in-time metrics and updating a cache. The cache is 
> then used to service all live REST requests. 
> - On the first request to a resource, a cache miss occurs and no data is 
> returned. I think this is acceptable since metrics take a few moments to 
> populate anyway right now. As the web client polls, the next request should 
> pickup the newly cached metrics.
> - Only URLs which are being asked for by incoming REST requests should be 
> considered for retrieval. After sometime,

Re: Review Request 47947: AMBARI-16922 Remove LogSearch dependency for Ranger

2016-05-27 Thread Gautam Borad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47947/#review135285
---


Ship it!




Ship It!

- Gautam Borad


On May 27, 2016, 10:07 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47947/
> ---
> 
> (Updated May 27, 2016, 10:07 a.m.)
> 
> 
> Review request for Ambari, Don Bosco Durai, Gautam Borad, Mahadev Konar, 
> Oliver Szabo, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-16922
> https://issues.apache.org/jira/browse/AMBARI-16922
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Removing Logsearch Dependency for Ranger Service
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  455fb67 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
>  e9b7e6f 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  539a57c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
>  2a3803b 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/metainfo.xml 
> a364418 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  f180aca 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
> 
> Diff: https://reviews.apache.org/r/47947/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger installation with and without Logsearch on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 47746: Update Moment.js to latest stable version 2.13.0

2016-05-27 Thread Sangeeta Ravindran



> On May 24, 2016, 12:25 a.m., Jaimin Jetly wrote:
> > Sangeeta Ravindran, Lets please add minified version of moment.js for 
> > 2.13.0 version.

Hello Jaimin,

I have attached a new patch with the minified version of moment.js 2.13.0. Can 
you please review?

Thanks,
Sangeeta


- Sangeeta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47746/#review134482
---


On May 23, 2016, 11:50 p.m., Sangeeta Ravindran wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47746/
> ---
> 
> (Updated May 23, 2016, 11:50 p.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly and Yusaku Sako.
> 
> 
> Bugs: ambari-16017
> https://issues.apache.org/jira/browse/ambari-16017
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The latest stable version of Moment.js is 2.13.0.
> This task is for updating the version of Moment.js used in
> 
> contrib/views/slider/src/main/resources/ui/vendor/scripts/common/
> ambari-web/vendor/scripts
> 
> 
> Diffs
> -
> 
>   ambari-web/vendor/scripts/moment.js fe0e19c 
>   contrib/views/slider/src/main/resources/ui/vendor/scripts/common/moment.js 
> b40374b 
> 
> Diff: https://reviews.apache.org/r/47746/diff/
> 
> 
> Testing
> ---
> 
> Manual testing - setting timezone in User Settings.
> Slider view - application start time.
> 
> Tests ran clean for ambari-web 
> 
> 27834 tests complete (42 seconds)
>   154 tests pending
> 
> Tests ran clean for contrib/views/slider
> Took 9506ms to run 345 tests. 345 passed, 0 failed.
> 
> 
> Thanks,
> 
> Sangeeta Ravindran
> 
>

Re: Review Request 47885: AMBARI-16894: Default Ranger repos for some services are not getting created

2016-05-27 Thread Srimanth Gunturi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47885/#review135279
---


Ship it!




Ship It!

- Srimanth Gunturi


On May 27, 2016, 4:30 a.m., Gautam Borad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47885/
> ---
> 
> (Updated May 27, 2016, 4:30 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, Jayush Luniya, 
> Mahadev Konar, Sumit Mohanty, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-16894
> https://issues.apache.org/jira/browse/AMBARI-16894
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fixed the logic of default repo creation that was broken due to previous 
> commits.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/ranger_functions_v2.py
>  efb0819 
> 
> Diff: https://reviews.apache.org/r/47885/diff/
> 
> 
> Testing
> ---
> 
> Tested on a local instance with different services.
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 47962: clean up import * for STORM, TEZ and ZEPPELIN services

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47962/#review135277
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 27, 2016, 4:28 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47962/
> ---
> 
> (Updated May 27, 2016, 4:28 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.
> 
> 
> Bugs: AMBARI-16848
> https://issues.apache.org/jira/browse/AMBARI-16848
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Python code at at common-services level used generic imports form 
> resource_management (from resource_management import *)
> Ideally, for easier code tracking and performance, these import should be 
> more specific, such as: 
> from resource_management.libraries.script.script import Script
> from resource_management.core.resources.system import Directory
> This patch cleans up import * from resource_management for PIG service 
> scripts in common-servicesPython code at at common-services level used 
> generic imports form resource_management (from resource_management import *)
> Ideally, for easier code tracking and performance, these import should be 
> more specific, such as: 
> from resource_management.libraries.script.script import Script
> from resource_management.core.resources.system import Directory
> This patch cleans up import * from resource_management for STORM, TEZ and 
> ZEPPELIN service scripts in common-services
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_windows.py
>  88e6246 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
>  198674d 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/service_check.py
>  5dfdcad 
> 
> Diff: https://reviews.apache.org/r/47962/diff/
> 
> 
> Testing
> ---
> 
> Passed mvn clean test -DskipSurefireTests
> STORM, TEZ and ZEPPELIN  fresh installation and service check
> 
> Hadoop QA added a comment - Yesterday
> -1 overall. Here are the results of testing the latest attachment 
> http://issues.apache.org/jira/secure/attachment/12806417/AMBARI-16848.patch
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> -1 tests included. The patch doesn't appear to include any new or modified 
> tests.
> Please justify why no new tests are needed for this patch.
> Also please list what manual steps were performed to verify this patch.
> +1 javac. The applied patch does not increase the total number of javac 
> compiler warnings.
> +1 release audit. The applied patch does not increase the total number of 
> release audit warnings.
> +1 core tests. The patch passed unit tests in ambari-server.
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7000//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7000//console
> This message is automatically generated.
> 
> 
> Thanks,
> 
> Juanjo  Marron
> 
>

Re: Review Request 47884: Ambari install failure due to zeppelin service install bug

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47884/#review135276
---


Ship it!




Let me know if you need me to commit this on your behalf.

- Alejandro Fernandez


On May 27, 2016, 4:03 p.m., Renjith Kamath wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47884/
> ---
> 
> (Updated May 27, 2016, 4:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, DIPAYAN BHOWMICK, Jayush 
> Luniya, Pallav Kulshreshtha, Rohit Choudhary, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-16893
> https://issues.apache.org/jira/browse/AMBARI-16893
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Service install fails on CentOS 6.6 due to zeppelin permission bug. Only 
> reproducible on 6.6
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  9cd1b1d 
> 
> Diff: https://reviews.apache.org/r/47884/diff/
> 
> 
> Testing
> ---
> 
> manually tested on CentOS 6.6 and 6.4
> 
> 
> Thanks,
> 
> Renjith Kamath
> 
>

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley



> On May 27, 2016, 1:45 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/services/MetricsRetrievalService.java,
> >  lines 138-145
> > 
> >
> > Doesn't seem like you need 2 of these that do the same thing.  REST vs 
> > JMX doesn't appear to matter.

Agreed - they are fully qualified URLs - I can have 1. Will change.


> On May 27, 2016, 1:45 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/services/MetricsRetrievalService.java,
> >  lines 395-401
> > 
> >
> > nit: these don't throw anything

LOG could throw somemthing. Also, System.currentTimeMillis() probably threw a 
Y2K exception. It could do it again in another billion years or so. But, you're 
right. I'll move them out.


> On May 27, 2016, 1:45 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/services/MetricsRetrievalService.java,
> >  lines 430-432
> > 
> >
> > nit: you have two Runnables that do nearly the identical thing except 
> > how to parse the resulting InputStream.  Could push most of the run() to 
> > MetricsRunnable and just have your subclasses parse.

Taking a look at this now ... I did the JMX one first, and when things were 
working, I did the REST one. Let me see if I can combine some logic.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135268
---


On May 27, 2016, 1:27 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 1:27 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the

Re: Review Request 47955: AMBARI-16906 Express upgrade: Oozie failed to start when user name and group are none-default values

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47955/#review135275
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 27, 2016, 2:27 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47955/
> ---
> 
> (Updated May 27, 2016, 2:27 p.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-16906
> https://issues.apache.org/jira/browse/AMBARI-16906
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> oozie_server_upgrade.py has hardcoded user name to oozie and group to hadoop. 
> Both values should have been pulled from params.py
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
>  a1f3336 
> 
> Diff: https://reviews.apache.org/r/47955/diff/
> 
> 
> Testing
> ---
> 
> patch a trunk cluster (Oozie use is non-default myoozie, group is non-default 
> myhadoop) with the change, run express upgrade, verify that with the change, 
> oozie restarted successfully during the EU.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 47941: [AMBARI-16920] Spark2 thrift server can not started due to miss of spark-thrift-fairscheduler.xml

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47941/#review135273
---




ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 (line 63)


Why is the tarball source coming from /tmp?


- Alejandro Fernandez


On May 27, 2016, 3:33 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47941/
> ---
> 
> (Updated May 27, 2016, 3:33 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-16920
> https://issues.apache.org/jira/browse/AMBARI-16920
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This a followup ticket for add spark2 stack definition. There's serveral 
> issues:
> 1.  Spark2 thrift server can not started due to miss of 
> spark-thrift-fairscheduler.xml
> 2.  Miss of add spark2 cache file in copy_barball.py
> 3.  Miss the role_commnad_order of spark2
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  286df8d 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
>  ded9959 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py
>  2eae3e7 
>   ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 
> 4a7c1d2 
> 
> Diff: https://reviews.apache.org/r/47941/diff/
> 
> 
> Testing
> ---
> 
> Manually verified.
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135268
---


Ship it!





ambari-server/src/main/java/org/apache/ambari/server/state/services/MetricsRetrievalService.java
 (lines 138 - 145)


Doesn't seem like you need 2 of these that do the same thing.  REST vs JMX 
doesn't appear to matter.



ambari-server/src/main/java/org/apache/ambari/server/state/services/MetricsRetrievalService.java
 (lines 395 - 401)


nit: these don't throw anything



ambari-server/src/main/java/org/apache/ambari/server/state/services/MetricsRetrievalService.java
 (lines 430 - 432)


nit: you have two Runnables that do nearly the identical thing except how 
to parse the resulting InputStream.  Could push most of the run() to 
MetricsRunnable and just have your subclasses parse.


- Nate Cole


On May 27, 2016, 1:27 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 1:27 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the request.
> 
> There are two inherent flaws with this approach:
> 
> - Even with millisecond JMX response times, multiplying this by 100's and 
> then adding parsing overhead causes a noticeable delay in the web client as 
> the federated requests are blocking the main UX request
> 
> - Although there is a threadpool which scales up to service these requests - 
> that only really works for 1 user. With multiple users logged in, you'd need 
> 100's upon 100's of threads pulling in the same JMX data
> 
> This data should never be queried for directly as part of the incoming REST 
> requests. Instead, an autonomous pool of threads should be constantly 
> retrieving

Re: Review Request 47914: AMBARI-16910. Hive Server Interactive. Change the timeout to 120 secs for LLAP alert command.

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47914/#review135272
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 27, 2016, 12:49 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47914/
> ---
> 
> (Updated May 27, 2016, 12:49 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-16910
> https://issues.apache.org/jira/browse/AMBARI-16910
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - Right now the timeout is 15 secs for LLAP alert.
> - **Reason to  increase:** Right now the command timeouts, because 15 secs is 
> less for it to complete depending on the cluster.
> - 
> - - Further, modified the messages displayed as part of alert.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/alerts.json 
> 0fad732 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_llap_app_status.py
>  b18c366 
> 
> Diff: https://reviews.apache.org/r/47914/diff/
> 
> 
> Testing
> ---
> 
> - Python UT passes.
> 
> 
> File Attachments
> 
> 
> NOT RUNNING state
>   
> https://reviews.apache.org/media/uploaded/files/2016/05/27/3579b990-22fd-41cc-8d3b-1c9d10d79a23__Screen_Shot_2016-05-26_at_5.34.45_PM.png
> RUNNING state
>   
> https://reviews.apache.org/media/uploaded/files/2016/05/27/b9c22854-8c5f-463c-bc3d-e1b38ea7c351__Screen_Shot_2016-05-26_at_5.32.15_PM.png
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 47923: clean up import * for SPARK service scripts in common-services

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47923/#review135270
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 26, 2016, 11:04 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47923/
> ---
> 
> (Updated May 26, 2016, 11:04 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.
> 
> 
> Bugs: AMBARI-16797
> https://issues.apache.org/jira/browse/AMBARI-16797
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Python code at at common-services level used generic imports form 
> resource_management (from resource_management import *)
> Ideally, for easier code tracking and performance, these import should be 
> more specific, such as: 
> from resource_management.libraries.script.script import Script
> from resource_management.core.resources.system import Directory
> This patch cleans up import * from resource_management for SPARK service 
> scripts in common-services
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/job_history_server.py
>  bccd714 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  c5f3eb6 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/service_check.py
>  694f046 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  eca8534 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  3838061 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py
>  9311454 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/status_params.py
>  86e7f7d 
> 
> Diff: https://reviews.apache.org/r/47923/diff/
> 
> 
> Testing
> ---
> 
> Passed mvn clean test -DskipSurefireTests
> PIG fresh installation and service check
> 
> Hadoop QA added a comment - 1 hour ago
> -1 overall. Here are the results of testing the latest attachment 
> http://issues.apache.org/jira/secure/attachment/12805785/AMBARI-16797.patch
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> -1 tests included. The patch doesn't appear to include any new or modified 
> tests.
> Please justify why no new tests are needed for this patch.
> Also please list what manual steps were performed to verify this patch.
> +1 javac. The applied patch does not increase the total number of javac 
> compiler warnings.
> +1 release audit. The applied patch does not increase the total number of 
> release audit warnings.
> +1 core tests. The patch passed unit tests in ambari-server.
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/6981//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/6981//console
> This message is automatically generated.
> 
> 
> Thanks,
> 
> Juanjo  Marron
> 
>

Re: Review Request 47965: clean up import * for RANGER service

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47965/#review135265
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 27, 2016, 4:41 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47965/
> ---
> 
> (Updated May 27, 2016, 4:41 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.
> 
> 
> Bugs: AMBARI-16743
> https://issues.apache.org/jira/browse/AMBARI-16743
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Python code at at common-services level used generic imports form 
> resource_management (from resource_management import *)
> Ideally, for easier code tracking and performance, these import should be 
> more specific, such as: 
> from resource_management.libraries.script.script import Script
> from resource_management.core.resources.system import Directory
> This patch cleans up import * from resource_management for RANGER service 
> scripts in common-services
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_service.py
>  2c7bd3c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger.py
>  3dc4914 
> 
> Diff: https://reviews.apache.org/r/47965/diff/
> 
> 
> Testing
> ---
> 
> Hadoop QA added a comment - 3 hours ago
> -1 overall. Here are the results of testing the latest attachment 
> http://issues.apache.org/jira/secure/attachment/12806494/AMBARI-16743.patch
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> -1 tests included. The patch doesn't appear to include any new or modified 
> tests.
> Please justify why no new tests are needed for this patch.
> Also please list what manual steps were performed to verify this patch.
> +1 javac. The applied patch does not increase the total number of javac 
> compiler warnings.
> +1 release audit. The applied patch does not increase the total number of 
> release audit warnings.
> +1 core tests. The patch passed unit tests in ambari-server.
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7015//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7015//console
> This message is automatically generated.
> 
> 
> Thanks,
> 
> Juanjo  Marron
> 
>

Re: Review Request 47964: clean up import * for ZOOKEEPER service

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47964/#review135266
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 27, 2016, 4:36 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47964/
> ---
> 
> (Updated May 27, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.
> 
> 
> Bugs: AMBARI-16850
> https://issues.apache.org/jira/browse/AMBARI-16850
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Python code at at common-services level used generic imports form 
> resource_management (from resource_management import *)
> Ideally, for easier code tracking and performance, these import should be 
> more specific, such as: 
> from resource_management.libraries.script.script import Script
> from resource_management.core.resources.system import Directory
> This patch cleans up import * from resource_management for ZOOKEEPER service 
> scripts in common-services
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_windows.py
>  c36e152 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/service_check.py
>  622a5eb 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper.py
>  e0ba54b 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_client.py
>  de4d6e1 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_service.py
>  9f943d7 
> 
> Diff: https://reviews.apache.org/r/47964/diff/
> 
> 
> Testing
> ---
> 
> Passed mvn clean test -DskipSurefireTests
> ZOOKEEPER fresh installation and service check
> 
> Hadoop QA added a comment - Yesterday
> -1 overall. Here are the results of testing the latest attachment 
> http://issues.apache.org/jira/secure/attachment/12806418/AMBARI-16850.patch
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> -1 tests included. The patch doesn't appear to include any new or modified 
> tests.
> Please justify why no new tests are needed for this patch.
> Also please list what manual steps were performed to verify this patch.
> +1 javac. The applied patch does not increase the total number of javac 
> compiler warnings.
> +1 release audit. The applied patch does not increase the total number of 
> release audit warnings.
> +1 core tests. The patch passed unit tests in ambari-server.
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7006//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7006//console
> This message is automatically generated.
> 
> 
> Thanks,
> 
> Juanjo  Marron
> 
>

Re: Review Request 47949: AMBARI-16923. Fix for getting the 'hive.llap.daemon.queue.name' config Property Attributes updated if there is a change in 'capacity-scheduler'.

2016-05-27 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47949/#review135264
---


Ship it!




Ship It!

- Alejandro Fernandez


On May 27, 2016, 4:39 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47949/
> ---
> 
> (Updated May 27, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Daniel Gergely, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-16923
> https://issues.apache.org/jira/browse/AMBARI-16923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - We need to read 'capacity-schdeuler' :
>1. Firstly, from configurations where the configs can either be a 
> dictionary or single "\n" string. It will be there if there is a change done 
> to it as part of current invocation. If they are not there, then go ahead and 
> read:
>2. services for the configs where again it can either be a a dictionary or 
> single "\n" string.
>   
>   
> - Removed stack_advisor.py 221-234 lines as they are no more valid as we dont 
> do calulation based on change to cache value now.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 0066e1d 
> 
> Diff: https://reviews.apache.org/r/47949/diff/
> 
> 
> Testing
> ---
> 
> - Python UT added.
> - 
> - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

(Updated May 27, 2016, 1:27 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Changes
---

Something is wrong with the ReviewBoard and the patch .. trying it again. Some 
files are not showing up


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab

Re: Review Request 47933: Add conditional constraints for Kerberos identities to control when they are created

2016-05-27 Thread Robert Levas



> On May 27, 2016, 12:32 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/AndPredicate.java,
> >  lines 19-24
> > 
> >
> > Seems like these are more generic than just for kerberos - maybe should 
> > go in org.apache.ambari.collections or something?  Is there no other 
> > guice/guava library doing the same stuff (json query framework or 
> > something)?

I Googled around and couldn't find anything other than 
`org.apache.commons.collections` stuff.  And since we already depend on the 
library it seemed like a good choice. 

I thought about moving this out but didnt have a good place the put it..  I can 
move to `org.apache.ambari.collections` or maybe 
`org.apache.ambari.server.collections` and sort of mirror what 
`org.apache.commons.collections` is doing.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47933/#review135255
---


On May 27, 2016, 6:01 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47933/
> ---
> 
> (Updated May 27, 2016, 6:01 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Oliver Szabo, Sandor 
> Magyari, Sumit Mohanty, and Swapan Shridhar.
> 
> 
> Bugs: AMBARI-16437
> https://issues.apache.org/jira/browse/AMBARI-16437
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add conditional constraints for Kerberos identities to control when they are 
> created. For example if Kerberos Identity should only be created (and 
> distributed) for a component when some other component or service is 
> installed. 
> 
> An example of this would be
> ```
> {
>   "name": "/HIVE/HIVE_SERVER/hive_server_hive",
>   "principal": {
> "configuration": 
> "hive-interactive-site/hive.llap.daemon.service.principal"
>   },
>   "keytab": {
> "configuration": "hive-interactive-site/hive.llap.daemon.keytab.file"
>   },
>   "when" : {
>   "contains" : ["services", "HIVE"]
>   }
> }
> ```
> 
> Note the "`when`" clause. This indicates that this identity should only be 
> processed when the set of services contains "HIVE".  An alternative to this 
> would be to test the set of components for a certain component.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  c67c55d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  0dbd357 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
>  bb2ed1c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
>  d31dd21 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/AndPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/ContainsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/ContextTransformer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/DelegatedMultiplePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/DelegatedSinglePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/EqualsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/NotPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/OperationPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/OrPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/Predicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/PredicateClassFactory.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/PredicateUtils.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  5393fd6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
>  d80d7cc 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
>  0ea7b26 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/AndPredicateTest.java
>

Re: Review Request 47949: AMBARI-16923. Fix for getting the 'hive.llap.daemon.queue.name' config Property Attributes updated if there is a change in 'capacity-scheduler'.

2016-05-27 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47949/#review135263
---


Ship it!




Ship It!

- Sumit Mohanty


On May 27, 2016, 4:39 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47949/
> ---
> 
> (Updated May 27, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Daniel Gergely, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-16923
> https://issues.apache.org/jira/browse/AMBARI-16923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - We need to read 'capacity-schdeuler' :
>1. Firstly, from configurations where the configs can either be a 
> dictionary or single "\n" string. It will be there if there is a change done 
> to it as part of current invocation. If they are not there, then go ahead and 
> read:
>2. services for the configs where again it can either be a a dictionary or 
> single "\n" string.
>   
>   
> - Removed stack_advisor.py 221-234 lines as they are no more valid as we dont 
> do calulation based on change to cache value now.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 0066e1d 
> 
> Diff: https://reviews.apache.org/r/47949/diff/
> 
> 
> Testing
> ---
> 
> - Python UT added.
> - 
> - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

(Updated May 27, 2016, 1:24 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 617553b

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

(Updated May 27, 2016, 1:22 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Changes
---

Noticed that when remote services were down, the MetricsRetrievalService was 
spamming the logs.


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/#review135261
---


Ship it!




Ship It!

- Nate Cole


On May 27, 2016, 12:26 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47961/
> ---
> 
> (Updated May 27, 2016, 12:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, 
> Robert Nettleton, and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16913
> https://issues.apache.org/jira/browse/AMBARI-16913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Incoming requests from the web client (or from any REST API) will eventually 
> be routed to the property provider / subresource framework. It is here were 
> any JMX data is queried for within the context of the REST request. In large 
> clusters, these requests can backup quite easily (even with a massive 
> threadpool), causing UX degradations in the web client:
> 
> ```
> Thread [qtp-ambari-client-38]
>   
> JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
>  Request, Predicate) line: 168   
>   JMXPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 156  
>   StackDefinedPropertyProvider.populateResources(Set, Request, 
> Predicate) line: 200 
>   ClusterControllerImpl.populateResources(Type, Set, Request, 
> Predicate) line: 155  
>   QueryImpl.queryForResources() line: 407 
>   QueryImpl.execute() line: 217   
>   ReadHandler.handleRequest(Request) line: 69 
>   GetRequest(BaseRequest).process() line: 145 
> ```
> 
> Consider one of the calls made by the web client:
> ```
> GET api/v1/clusters/c1/components/?
> ServiceComponentInfo/category=MASTER&
> fields=
> ServiceComponentInfo/service_name,
> host_components/HostRoles/display_name,
> host_components/HostRoles/host_name,
> host_components/HostRoles/state,
> host_components/HostRoles/maintenance_state,
> host_components/HostRoles/stale_configs,
> host_components/HostRoles/ha_state,
> host_components/HostRoles/desired_admin_state,
> host_components/metrics/jvm/memHeapUsedM,
> host_components/metrics/jvm/HeapMemoryMax,
> host_components/metrics/jvm/HeapMemoryUsed,
> host_components/metrics/jvm/memHeapCommittedM,
> host_components/metrics/mapred/jobtracker/trackers_decommissioned,
> host_components/metrics/cpu/cpu_wio,
> host_components/metrics/rpc/client/RpcQueueTime_avg_time,
> host_components/metrics/dfs/FSNamesystem/*,
> host_components/metrics/dfs/namenode/Version,
> host_components/metrics/dfs/namenode/LiveNodes,
> host_components/metrics/dfs/namenode/DeadNodes,
> host_components/metrics/dfs/namenode/DecomNodes,
> host_components/metrics/dfs/namenode/TotalFiles,
> host_components/metrics/dfs/namenode/UpgradeFinalized,
> host_components/metrics/dfs/namenode/Safemode,
> host_components/metrics/runtime/StartTime
> ```
> 
> This query is essentially saying that for every {{MASTER}}, get metrics from 
> them. The problem is that in a large cluster, there could be 100 masters, yet 
> the metrics being asked for are only for NameNode. As a result, the JMX 
> endpoints for all 100 masters are queried - *live* - as part of the request.
> 
> There are two inherent flaws with this approach:
> 
> - Even with millisecond JMX response times, multiplying this by 100's and 
> then adding parsing overhead causes a noticeable delay in the web client as 
> the federated requests are blocking the main UX request
> 
> - Although there is a threadpool which scales up to service these requests - 
> that only really works for 1 user. With multiple users logged in, you'd need 
> 100's upon 100's of threads pulling in the same JMX data
> 
> This data should never be queried for directly as part of the incoming REST 
> requests. Instead, an autonomous pool of threads should be constantly 
> retrieving these point-in-time metrics and updating a cache. The cache is 
> then used to service all live REST requests. 
> - On the first request to a resource, a cache miss occurs and no data is 
> returned. I think this is acceptable since metrics take a few moments to 
> populate anyway right now. As the web client polls, the next request should 
> pickup the newly cached metrics.
> - Only URLs which are being asked for by incoming REST requests should be 
> considered for retrieval. After sometime, if they haven't been requested, 
> then the headless threadpool can stop trying to update their data
> - All JMX data will be parsed and stored in-memory, in an expiring cache
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
> 186e272 
>   
>

Review Request 47967: Atlas server start failed after Ambari upgrade due to missing solrCloudCli.sh script

2016-05-27 Thread Tom Beerbower


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47967/
---

Review request for Ambari, John Speidel and Robert Levas.


Bugs: AMBARI-16932
https://issues.apache.org/jira/browse/AMBARI-16932


Repository: ambari


Description
---

Steps

1.Deploy HDP-2.4.2.0 cluster with Ambari 2.2.2.0 (including Atlas)
2.Upgrade Ambari to 2.4.0.0
3.Stop and start all services

Result

Atlas Metadata server start fails with below error:
  Traceback (most recent call last):
File 
"/var/lib/ambari-agent/cache/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py",
 line 165, in 
  MetadataServer().execute()
  raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'export 
JAVA_HOME=/usr/jdk64/jdk1.7.0_67 ; 
/usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh -z 
os-r6-tgvuks-dgm10toeriedwngdha-r6-4.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-3.openstacklocal:2181,os-r6-tgvuks-dgm10toeriedwngdha-r6-2.openstacklocal:2181None
 --upload-config -d 
/usr/lib/ambari-logsearch-solr/server/solr/configsets/basic_configs/conf -cs 
basic_configs -rt 5 -i 10' returned 127. -bash: 
/usr/lib/ambari-logsearch-solr-client/solrCloudCli.sh: No such file or directory


Diffs
-

  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 6f287dc 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 3a9c614 
  ambari-server/src/test/python/stacks/2.3/configs/default.json ef04248 
  ambari-server/src/test/python/stacks/2.3/configs/secure.json 166fd6f 
  ambari-server/src/test/python/stacks/2.5/configs/default.json 0077280 

Diff: https://reviews.apache.org/r/47967/diff/


Testing
---

Manual test

Adjust unit tests

mvn clean test


Thanks,

Tom Beerbower

Re: Review Request 47915: Oozie and Hive Server start fail during EU with missing DB class exception

2016-05-27 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47915/#review135259
---


Ship it!




Ship It!

- Andrew Onischuk


On May 27, 2016, 4:47 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47915/
> ---
> 
> (Updated May 27, 2016, 4:47 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-16912
> https://issues.apache.org/jira/browse/AMBARI-16912
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Steps
> Deploy HDP 2.4.0.0 cluster with Ambari 2.2.1.1 (Secure, HA cluster, mysql DB)
> Upgrade Ambari to 2.4.0.0
> Perform Express Upgrade to HDP-2.5.0.0-555
> Result
> Errors seen during start of Hive Server and Oozie
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/dbConfiguration.py c92e580 
>   ambari-server/src/main/python/ambari_server/serverSetup.py f01465d 
>   ambari-server/src/main/python/ambari_server/serverUpgrade.py cdcce1c 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
>  a1ffc5e 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.py
>  716b612 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
>  2f091a5 
>   
> ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py
>  0a39219 
> 
> Diff: https://reviews.apache.org/r/47915/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 47915: Oozie and Hive Server start fail during EU with missing DB class exception

2016-05-27 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47915/
---

(Updated Травень 27, 2016, 4:47 після полудня)


Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Sumit 
Mohanty.


Bugs: AMBARI-16912
https://issues.apache.org/jira/browse/AMBARI-16912


Repository: ambari


Description
---

Steps
Deploy HDP 2.4.0.0 cluster with Ambari 2.2.1.1 (Secure, HA cluster, mysql DB)
Upgrade Ambari to 2.4.0.0
Perform Express Upgrade to HDP-2.5.0.0-555
Result
Errors seen during start of Hive Server and Oozie


Diffs (updated)
-

  ambari-server/src/main/python/ambari_server/dbConfiguration.py c92e580 
  ambari-server/src/main/python/ambari_server/serverSetup.py f01465d 
  ambari-server/src/main/python/ambari_server/serverUpgrade.py cdcce1c 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
 a1ffc5e 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.py
 716b612 
  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
 2f091a5 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py
 0a39219 

Diff: https://reviews.apache.org/r/47915/diff/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Review Request 47965: clean up import * for RANGER service

2016-05-27 Thread Juanjo Marron


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47965/
---

Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.


Bugs: AMBARI-16743
https://issues.apache.org/jira/browse/AMBARI-16743


Repository: ambari


Description
---

Python code at at common-services level used generic imports form 
resource_management (from resource_management import *)
Ideally, for easier code tracking and performance, these import should be more 
specific, such as: 
from resource_management.libraries.script.script import Script
from resource_management.core.resources.system import Directory
This patch cleans up import * from resource_management for RANGER service 
scripts in common-services


Diffs
-

  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_service.py
 2c7bd3c 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger.py
 3dc4914 

Diff: https://reviews.apache.org/r/47965/diff/


Testing
---

Hadoop QA added a comment - 3 hours ago
-1 overall. Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12806494/AMBARI-16743.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac 
compiler warnings.
+1 release audit. The applied patch does not increase the total number of 
release audit warnings.
+1 core tests. The patch passed unit tests in ambari-server.
Test results: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7015//testReport/
Console output: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7015//console
This message is automatically generated.


Thanks,

Juanjo  Marron

Re: Review Request 47949: AMBARI-16923. Fix for getting the 'hive.llap.daemon.queue.name' config Property Attributes updated if there is a change in 'capacity-scheduler'.

2016-05-27 Thread Swapan Shridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47949/
---

(Updated May 27, 2016, 4:39 p.m.)


Review request for Ambari, Alejandro Fernandez, Daniel Gergely, and Sumit 
Mohanty.


Changes
---

Updated review based on Daniel's comment.


Bugs: AMBARI-16923
https://issues.apache.org/jira/browse/AMBARI-16923


Repository: ambari


Description
---

- We need to read 'capacity-schdeuler' :
   1. Firstly, from configurations where the configs can either be a dictionary 
or single "\n" string. It will be there if there is a change done to it as part 
of current invocation. If they are not there, then go ahead and read:
   2. services for the configs where again it can either be a a dictionary or 
single "\n" string.
  
  
- Removed stack_advisor.py 221-234 lines as they are no more valid as we dont 
do calulation based on change to cache value now.


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
8c5351f 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 0066e1d 

Diff: https://reviews.apache.org/r/47949/diff/


Testing
---

- Python UT added.
- 
- Python UT passes.


Thanks,

Swapan Shridhar

Review Request 47964: clean up import * for ZOOKEEPER service

2016-05-27 Thread Juanjo Marron


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47964/
---

Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.


Bugs: AMBARI-16850
https://issues.apache.org/jira/browse/AMBARI-16850


Repository: ambari


Description
---

Python code at at common-services level used generic imports form 
resource_management (from resource_management import *)
Ideally, for easier code tracking and performance, these import should be more 
specific, such as: 
from resource_management.libraries.script.script import Script
from resource_management.core.resources.system import Directory
This patch cleans up import * from resource_management for ZOOKEEPER service 
scripts in common-services


Diffs
-

  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_windows.py
 c36e152 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/service_check.py
 622a5eb 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper.py
 e0ba54b 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_client.py
 de4d6e1 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_service.py
 9f943d7 

Diff: https://reviews.apache.org/r/47964/diff/


Testing
---

Passed mvn clean test -DskipSurefireTests
ZOOKEEPER fresh installation and service check

Hadoop QA added a comment - Yesterday
-1 overall. Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12806418/AMBARI-16850.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac 
compiler warnings.
+1 release audit. The applied patch does not increase the total number of 
release audit warnings.
+1 core tests. The patch passed unit tests in ambari-server.
Test results: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7006//testReport/
Console output: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7006//console
This message is automatically generated.


Thanks,

Juanjo  Marron

Re: Review Request 47949: AMBARI-16923. Fix for getting the 'hive.llap.daemon.queue.name' config Property Attributes updated if there is a change in 'capacity-scheduler'.

2016-05-27 Thread Swapan Shridhar



> On May 27, 2016, 12:09 p.m., Daniel Gergely wrote:
> >
> 
> Daniel Gergely wrote:
> Just a quiestion: if \n separated string is needed only for the UI, 
> wouldnt it be better to handle it on the UI side? So UI can do the 
> conversion, since it is not needed anywhere else. If it is also used 
> somewhere else, then fine.

Yes. The UI and BP both send capacity-schdeuler configs as dictionary in first 
call. But with this change, we will be able to cater both the worlds as a 
backend.


> On May 27, 2016, 12:09 p.m., Daniel Gergely wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, 
> > lines 989-990
> > 
> >
> > You can use cap_sched_props_as_dict here as well

Done.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47949/#review135224
---


On May 27, 2016, 10:48 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47949/
> ---
> 
> (Updated May 27, 2016, 10:48 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Daniel Gergely, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-16923
> https://issues.apache.org/jira/browse/AMBARI-16923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - We need to read 'capacity-schdeuler' :
>1. Firstly, from configurations where the configs can either be a 
> dictionary or single "\n" string. It will be there if there is a change done 
> to it as part of current invocation. If they are not there, then go ahead and 
> read:
>2. services for the configs where again it can either be a a dictionary or 
> single "\n" string.
>   
>   
> - Removed stack_advisor.py 221-234 lines as they are no more valid as we dont 
> do calulation based on change to cache value now.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 0066e1d 
> 
> Diff: https://reviews.apache.org/r/47949/diff/
> 
> 
> Testing
> ---
> 
> - Python UT added.
> - 
> - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Review Request 47963: clean up import * for YARN service

2016-05-27 Thread Juanjo Marron


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47963/
---

Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.


Bugs: AMBARI-16849
https://issues.apache.org/jira/browse/AMBARI-16849


Repository: ambari


Description
---

Python code at at common-services level used generic imports form 
resource_management (from resource_management import *)
Ideally, for easier code tracking and performance, these import should be more 
specific, such as: 
from resource_management.libraries.script.script import Script
from resource_management.core.resources.system import Directory
This patch cleans up import * from resource_management for YARN service scripts 
in common-services


Diffs
-

  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py
 4ec6aa7 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py
 34c683a 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/install_jars.py
 44015bf 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapred_service_check.py
 13172ee 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
 3ea3d75 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
 7be49d5 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager_upgrade.py
 1c886f9 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_windows.py
 0f8ce73 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py
 b1179b9 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py
 792a681 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
 0e2c519 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
 4d65a40 

Diff: https://reviews.apache.org/r/47963/diff/


Testing
---

Passed mvn clean test -DskipSurefireTests
YARN fresh installation and service check

Hadoop QA added a comment - Yesterday
-1 overall. Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12806423/AMBARI-16849.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac 
compiler warnings.
+1 release audit. The applied patch does not increase the total number of 
release audit warnings.
+1 core tests. The patch passed unit tests in ambari-server.
Test results: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7005//testReport/
Console output: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7005//console
This message is automatically generated.
Reply


Thanks,

Juanjo  Marron

Re: Review Request 47933: Add conditional constraints for Kerberos identities to control when they are created

2016-05-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47933/#review135255
---




ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/AndPredicate.java
 (lines 19 - 24)


Seems like these are more generic than just for kerberos - maybe should go 
in org.apache.ambari.collections or something?  Is there no other guice/guava 
library doing the same stuff (json query framework or something)?


- Nate Cole


On May 27, 2016, 6:01 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47933/
> ---
> 
> (Updated May 27, 2016, 6:01 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Oliver Szabo, Sandor 
> Magyari, Sumit Mohanty, and Swapan Shridhar.
> 
> 
> Bugs: AMBARI-16437
> https://issues.apache.org/jira/browse/AMBARI-16437
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add conditional constraints for Kerberos identities to control when they are 
> created. For example if Kerberos Identity should only be created (and 
> distributed) for a component when some other component or service is 
> installed. 
> 
> An example of this would be
> ```
> {
>   "name": "/HIVE/HIVE_SERVER/hive_server_hive",
>   "principal": {
> "configuration": 
> "hive-interactive-site/hive.llap.daemon.service.principal"
>   },
>   "keytab": {
> "configuration": "hive-interactive-site/hive.llap.daemon.keytab.file"
>   },
>   "when" : {
>   "contains" : ["services", "HIVE"]
>   }
> }
> ```
> 
> Note the "`when`" clause. This indicates that this identity should only be 
> processed when the set of services contains "HIVE".  An alternative to this 
> would be to test the set of components for a certain component.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  c67c55d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  0dbd357 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
>  bb2ed1c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
>  d31dd21 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/AndPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/ContainsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/ContextTransformer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/DelegatedMultiplePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/DelegatedSinglePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/EqualsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/NotPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/OperationPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/OrPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/Predicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/PredicateClassFactory.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/PredicateUtils.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  5393fd6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
>  d80d7cc 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
>  0ea7b26 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/AndPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/ContainsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/ContextTransformerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/EqualsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/NotPredicateTest.java
>  PRE-CREATION 
>   
>

Re: Review Request 47933: Add conditional constraints for Kerberos identities to control when they are created

2016-05-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47933/#review135256
---


Ship it!




Ship It!

- Nate Cole


On May 27, 2016, 6:01 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47933/
> ---
> 
> (Updated May 27, 2016, 6:01 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Oliver Szabo, Sandor 
> Magyari, Sumit Mohanty, and Swapan Shridhar.
> 
> 
> Bugs: AMBARI-16437
> https://issues.apache.org/jira/browse/AMBARI-16437
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add conditional constraints for Kerberos identities to control when they are 
> created. For example if Kerberos Identity should only be created (and 
> distributed) for a component when some other component or service is 
> installed. 
> 
> An example of this would be
> ```
> {
>   "name": "/HIVE/HIVE_SERVER/hive_server_hive",
>   "principal": {
> "configuration": 
> "hive-interactive-site/hive.llap.daemon.service.principal"
>   },
>   "keytab": {
> "configuration": "hive-interactive-site/hive.llap.daemon.keytab.file"
>   },
>   "when" : {
>   "contains" : ["services", "HIVE"]
>   }
> }
> ```
> 
> Note the "`when`" clause. This indicates that this identity should only be 
> processed when the set of services contains "HIVE".  An alternative to this 
> would be to test the set of components for a certain component.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  c67c55d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  0dbd357 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
>  bb2ed1c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
>  d31dd21 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/AndPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/ContainsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/ContextTransformer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/DelegatedMultiplePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/DelegatedSinglePredicateContainer.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/EqualsPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/NotPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/OperationPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/OrPredicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/Predicate.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/PredicateClassFactory.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/eval/PredicateUtils.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  5393fd6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
>  d80d7cc 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptorTest.java
>  0ea7b26 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/AndPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/ContainsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/ContextTransformerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/EqualsPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/NotPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/OrPredicateTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/eval/PredicateUtilsTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47933/diff/
> 
> 
> Testing
> ---
> 
> Manually tested 
> 
> # Local test results: PENDING
> 
>

Review Request 47962: clean up import * for STORM, TEZ and ZEPPELIN services

2016-05-27 Thread Juanjo Marron


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47962/
---

Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Matt.


Bugs: AMBARI-16848
https://issues.apache.org/jira/browse/AMBARI-16848


Repository: ambari


Description
---

Python code at at common-services level used generic imports form 
resource_management (from resource_management import *)
Ideally, for easier code tracking and performance, these import should be more 
specific, such as: 
from resource_management.libraries.script.script import Script
from resource_management.core.resources.system import Directory
This patch cleans up import * from resource_management for PIG service scripts 
in common-servicesPython code at at common-services level used generic imports 
form resource_management (from resource_management import *)
Ideally, for easier code tracking and performance, these import should be more 
specific, such as: 
from resource_management.libraries.script.script import Script
from resource_management.core.resources.system import Directory
This patch cleans up import * from resource_management for STORM, TEZ and 
ZEPPELIN service scripts in common-services


Diffs
-

  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_windows.py
 88e6246 
  
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
 198674d 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/service_check.py
 5dfdcad 

Diff: https://reviews.apache.org/r/47962/diff/


Testing
---

Passed mvn clean test -DskipSurefireTests
STORM, TEZ and ZEPPELIN  fresh installation and service check

Hadoop QA added a comment - Yesterday
-1 overall. Here are the results of testing the latest attachment 
http://issues.apache.org/jira/secure/attachment/12806417/AMBARI-16848.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac 
compiler warnings.
+1 release audit. The applied patch does not increase the total number of 
release audit warnings.
+1 core tests. The patch passed unit tests in ambari-server.
Test results: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7000//testReport/
Console output: 
https://builds.apache.org/job/Ambari-trunk-test-patch/7000//console
This message is automatically generated.


Thanks,

Juanjo  Marron

Re: Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

(Updated May 27, 2016, 12:26 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 617553b

Re: Review Request 47946: Yarn minimum container size calculation problem in stack advisor

2016-05-27 Thread Swapan Shridhar



> On May 27, 2016, 11:05 a.m., Swapan Shridhar wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, 
> > line 643
> > 
> >
> > Should we do max here, or just pick configurations if 
> > 'yarn.scheduler.minimum-allocation-mb' value is there, as that's the latest 
> > value in terms of timeline compared to services. Your inputs please?
> 
> Daniel Gergely wrote:
> Basically I asked the same question. Services array might not needed at 
> all, since the output of the property calculation (for yarn) is in the 
> configurations array, so we can use it and that is the most accurate one. But 
> services is used at many other places, so I did not want to remove it 
> completely, because I might broke something.

Sure. How about the idea of look into configuration for a given config. If it 
exists, implies we changed/recommended it in current ST. Otherwise read from 
services where it should most probably be there. Otherwise raise.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47946/#review135215
---


On May 27, 2016, 10:14 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47946/
> ---
> 
> (Updated May 27, 2016, 10:14 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, Swapan Shridhar, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16921
> https://issues.apache.org/jira/browse/AMBARI-16921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Stack advisor gets yarn minimum container size property from only the 
> "services" input, but does not take into account the already calculated 
> configurations.
> As a result memory size for hive interactive server is not correct, so it 
> doesn't start.
> 
> Now services and configurations array are both considered. Does services even 
> needed here?
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
> 
> Diff: https://reviews.apache.org/r/47946/diff/
> 
> 
> Testing
> ---
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:45.484s
> [INFO] Finished at: Fri May 27 12:05:56 CEST 2016
> [INFO] Final Memory: 119M/797M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Review Request 47961: Web Client Requests Handled By Jetty Should Not Be Blocked By JMX Property Providers

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47961/
---

Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, Robert 
Nettleton, and Srimanth Gunturi.


Bugs: AMBARI-16913
https://issues.apache.org/jira/browse/AMBARI-16913


Repository: ambari


Description
---

Incoming requests from the web client (or from any REST API) will eventually be 
routed to the property provider / subresource framework. It is here were any 
JMX data is queried for within the context of the REST request. In large 
clusters, these requests can backup quite easily (even with a massive 
threadpool), causing UX degradations in the web client:

```
Thread [qtp-ambari-client-38]

JMXPropertyProvider(ThreadPoolEnabledPropertyProvider).populateResources(Set,
 Request, Predicate) line: 168   
JMXPropertyProvider.populateResources(Set, Request, 
Predicate) line: 156  
StackDefinedPropertyProvider.populateResources(Set, Request, 
Predicate) line: 200 
ClusterControllerImpl.populateResources(Type, Set, Request, 
Predicate) line: 155  
QueryImpl.queryForResources() line: 407 
QueryImpl.execute() line: 217   
ReadHandler.handleRequest(Request) line: 69 
GetRequest(BaseRequest).process() line: 145 
```

Consider one of the calls made by the web client:
```
GET api/v1/clusters/c1/components/?
ServiceComponentInfo/category=MASTER&
fields=
ServiceComponentInfo/service_name,
host_components/HostRoles/display_name,
host_components/HostRoles/host_name,
host_components/HostRoles/state,
host_components/HostRoles/maintenance_state,
host_components/HostRoles/stale_configs,
host_components/HostRoles/ha_state,
host_components/HostRoles/desired_admin_state,
host_components/metrics/jvm/memHeapUsedM,
host_components/metrics/jvm/HeapMemoryMax,
host_components/metrics/jvm/HeapMemoryUsed,
host_components/metrics/jvm/memHeapCommittedM,
host_components/metrics/mapred/jobtracker/trackers_decommissioned,
host_components/metrics/cpu/cpu_wio,
host_components/metrics/rpc/client/RpcQueueTime_avg_time,
host_components/metrics/dfs/FSNamesystem/*,
host_components/metrics/dfs/namenode/Version,
host_components/metrics/dfs/namenode/LiveNodes,
host_components/metrics/dfs/namenode/DeadNodes,
host_components/metrics/dfs/namenode/DecomNodes,
host_components/metrics/dfs/namenode/TotalFiles,
host_components/metrics/dfs/namenode/UpgradeFinalized,
host_components/metrics/dfs/namenode/Safemode,
host_components/metrics/runtime/StartTime
```

This query is essentially saying that for every {{MASTER}}, get metrics from 
them. The problem is that in a large cluster, there could be 100 masters, yet 
the metrics being asked for are only for NameNode. As a result, the JMX 
endpoints for all 100 masters are queried - *live* - as part of the request.

There are two inherent flaws with this approach:

- Even with millisecond JMX response times, multiplying this by 100's and then 
adding parsing overhead causes a noticeable delay in the web client as the 
federated requests are blocking the main UX request

- Although there is a threadpool which scales up to service these requests - 
that only really works for 1 user. With multiple users logged in, you'd need 
100's upon 100's of threads pulling in the same JMX data

This data should never be queried for directly as part of the incoming REST 
requests. Instead, an autonomous pool of threads should be constantly 
retrieving these point-in-time metrics and updating a cache. The cache is then 
used to service all live REST requests. 
- On the first request to a resource, a cache miss occurs and no data is 
returned. I think this is acceptable since metrics take a few moments to 
populate anyway right now. As the web client polls, the next request should 
pickup the newly cached metrics.
- Only URLs which are being asked for by incoming REST requests should be 
considered for retrieval. After sometime, if they haven't been requested, then 
the headless threadpool can stop trying to update their data
- All JMX data will be parsed and stored in-memory, in an expiring cache


Diffs
-

  ambari-server/src/main/java/org/apache/ambari/server/AmbariService.java 
186e272 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 99a6cab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 617553b 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractProviderModule.java

Re: Review Request 47960: Make it possible to debug ambari-agent in runtime to investigate memory leaks etc.

2016-05-27 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47960/#review135247
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Травень 27, 2016, 4:04 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47960/
> ---
> 
> (Updated Травень 27, 2016, 4:04 після полудня)
> 
> 
> Review request for Ambari and Dmitro Lisnichenko.
> 
> 
> Bugs: AMBARI-16934
> https://issues.apache.org/jira/browse/AMBARI-16934
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HeartbeatHandlers.py 67e3c77 
>   ambari-agent/src/main/python/ambari_agent/RemoteDebugUtils.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/debug.py PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47960/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 47960: Make it possible to debug ambari-agent in runtime to investigate memory leaks etc.

2016-05-27 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47960/
---

Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-16934
https://issues.apache.org/jira/browse/AMBARI-16934


Repository: ambari


Description
---


Diffs
-

  ambari-agent/src/main/python/ambari_agent/HeartbeatHandlers.py 67e3c77 
  ambari-agent/src/main/python/ambari_agent/RemoteDebugUtils.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/debug.py PRE-CREATION 

Diff: https://reviews.apache.org/r/47960/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 47884: Ambari install failure due to zeppelin service install bug

2016-05-27 Thread Renjith Kamath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47884/
---

(Updated May 27, 2016, 4:03 p.m.)


Review request for Ambari, Alejandro Fernandez, DIPAYAN BHOWMICK, Jayush 
Luniya, Pallav Kulshreshtha, Rohit Choudhary, and Sumit Mohanty.


Bugs: AMBARI-16893
https://issues.apache.org/jira/browse/AMBARI-16893


Repository: ambari


Description
---

Service install fails on CentOS 6.6 due to zeppelin permission bug. Only 
reproducible on 6.6


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
 9cd1b1d 

Diff: https://reviews.apache.org/r/47884/diff/


Testing
---

manually tested on CentOS 6.6 and 6.4


Thanks,

Renjith Kamath

Re: Review Request 47957: Add support for Ubuntu 16

2016-05-27 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47957/
---

(Updated May 27, 2016, 4 p.m.)


Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-16931
https://issues.apache.org/jira/browse/AMBARI-16931


Repository: ambari


Description (updated)
---

So, there are two parts here. One is just to functionally make sure that
Ambari can accept doing an installation on Ubuntu. The second is to run Ambari
through its paces to make sure things actually work on Ubuntu 16.


Diffs
-

  ambari-common/src/main/python/ambari_commons/resources/os_family.json 92d9934 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/metainfo.xml
 781dee2 
  ambari-server/src/main/resources/common-services/GANGLIA/3.5.0/metainfo.xml 
f830967 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
16d0a84 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
7fc478f 
  ambari-server/src/main/resources/common-services/KAFKA/0.8.1/metainfo.xml 
c1dbf43 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 7d50ba1 
  ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/metainfo.xml 
7d8959d 
  ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/metainfo.xml 
adb5572 
  ambari-server/src/main/resources/common-services/RANGER/0.4.0/metainfo.xml 
9c33630 
  ambari-server/src/main/resources/common-services/SPARK/1.2.1/metainfo.xml 
4cd681a 
  ambari-server/src/main/resources/common-services/SPARK2/2.0.0/metainfo.xml 
98f1696 
  ambari-server/src/main/resources/common-services/STORM/0.9.3/metainfo.xml 
ed4337c 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml 
dee25ff 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml 
0b35357 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml 
1e1f8c3 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml 
437036f 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml 
2c40e25 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml 
22f93fe 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml 
6653670 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml 
95d6e2d 
  
ambari-server/src/main/resources/stacks/HDP/2.3.ECS/services/ZOOKEEPER/metainfo.xml
 4833857 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml 
57b19df 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml 
0abb468 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
875966c 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml 
57ee0b8 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml 
bb26f12 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml 
5e5c5cf 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml
 5804f88 
  ambari-server/src/main/resources/stacks/HDP/2.5/repos/repoinfo.xml e5c53ce 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml 
09791e4 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ZEPPELIN/metainfo.xml 
d4e9e3a 
  ambari-server/src/main/resources/version_definition.xsd 1c57e90 
  contrib/version-builder/version_builder.py 415e940 

Diff: https://reviews.apache.org/r/47957/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 47783: Cluster operator and cluster admin not allowed to install ambari agent

2016-05-27 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47783/
---

(Updated May 27, 2016, 11:40 a.m.)


Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Nate 
Cole.


Changes
---

Fixed a unit test - 
`org.apache.ambari.server.controller.internal.RequestResourceProviderTest`

```
Running org.apache.ambari.server.controller.internal.RequestResourceProviderTest
Tests run: 39, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.144 sec - in 
org.apache.ambari.server.controller.internal.RequestResourceProviderTest

Results :

Tests run: 39, Failures: 0, Errors: 0, Skipped: 0
```


Bugs: AMBARI-16851
https://issues.apache.org/jira/browse/AMBARI-16851


Repository: ambari


Description
---

Cluster operator and the cluster admin must be allowed to add/delete hosts but 
install of agents using /bootstrap fails with 403


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
 5b318af 
  
ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinition.java
 1f189a3 
  
ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionManager.java
 97aa8a0 
  
ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionSpec.java
 2dfa1f2 
  
ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariAuthorizationFilter.java
 5c74f07 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 0deba5d 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 2c2d743 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql ee87cc5 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql edc46f7 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 6f38ec8 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
ca57de5 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 61aadf0 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 9269b13 
  
ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml
 6304baf 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 3ec9cb3 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ActionResourceProviderTest.java
 96995b4 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
 65efc63 
  
ambari-server/src/test/java/org/apache/ambari/server/customactions/ActionDefinitionManagerTest.java
 ec84922 
  
ambari-server/src/test/java/org/apache/ambari/server/security/TestAuthenticationFactory.java
 69b4b08 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
 4dedc98 
  
ambari-server/src/test/resources/custom_action_definitions/cust_action_definitions1.xml
 9cee575 
  
ambari-server/src/test/resources/custom_action_definitions_invalid/cust_action_definitions_invalid.xml
 PRE-CREATION 

Diff: https://reviews.apache.org/r/47783/diff/


Testing
---

Manually tested, newly created cluster and upgrade 

# Local test results:
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:15:22.164s
[INFO] Finished at: Tue May 24 13:28:21 EDT 2016
[INFO] Final Memory: 59M/1807M
[INFO] 

#Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 47783: Cluster operator and cluster admin not allowed to install ambari agent

2016-05-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47783/#review135246
---


Ship it!




Ship It!

- Jonathan Hurley


On May 27, 2016, 9:52 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47783/
> ---
> 
> (Updated May 27, 2016, 9:52 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-16851
> https://issues.apache.org/jira/browse/AMBARI-16851
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Cluster operator and the cluster admin must be allowed to add/delete hosts 
> but install of agents using /bootstrap fails with 403
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  5b318af 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinition.java
>  1f189a3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionManager.java
>  97aa8a0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionSpec.java
>  2dfa1f2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariAuthorizationFilter.java
>  5c74f07 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  0deba5d 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 2c2d743 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql ee87cc5 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql edc46f7 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 6f38ec8 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ca57de5 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 61aadf0 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 9269b13 
>   
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml
>  6304baf 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  3ec9cb3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ActionResourceProviderTest.java
>  96995b4 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  65efc63 
>   
> ambari-server/src/test/java/org/apache/ambari/server/customactions/ActionDefinitionManagerTest.java
>  ec84922 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/TestAuthenticationFactory.java
>  69b4b08 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  4dedc98 
>   
> ambari-server/src/test/resources/custom_action_definitions/cust_action_definitions1.xml
>  9cee575 
>   
> ambari-server/src/test/resources/custom_action_definitions_invalid/cust_action_definitions_invalid.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47783/diff/
> 
> 
> Testing
> ---
> 
> Manually tested, newly created cluster and upgrade 
> 
> # Local test results:
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:15:22.164s
> [INFO] Finished at: Tue May 24 13:28:21 EDT 2016
> [INFO] Final Memory: 59M/1807M
> [INFO] 
> 
> 
> #Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services

2016-05-27 Thread Tim Thorpe


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47656/
---

(Updated May 27, 2016, 3:21 p.m.)


Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh 
chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, 
and Yusaku Sako.


Changes
---

Repatched over latest trunk code


Bugs: AMBARI-12885
https://issues.apache.org/jira/browse/AMBARI-12885


Repository: ambari


Description
---

The purpose of this proposal is to facilitate adding custom services to an 
existing stack. Ideally this would support adding and upgrading custom services 
separately from the core services defined in the stack. In particular we are 
looking at custom services that need to support several different stacks 
(different distributions of Ambari). The release cycle of the custom services 
may be different from that of the core stack; that is, a custom service may be 
upgraded at a different rate than the core distribution itself and may be 
upgraded multiple times within the lifespan of a single release of the core 
distribution.

One possible approach to handling this would be dynamically extending a stack 
(after install time). It would be best to extend the stack in packages where a 
stack extension package can have one or more custom services.


Diffs (updated)
-

  ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 
fcea23f 
  ambari-agent/src/main/python/ambari_agent/FileCache.py b7c5dee 
  ambari-server/conf/unix/ambari.properties 9f1692e 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
 cf7c391 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 c54fe3f 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java
 557ce98 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 f4a615c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkRequest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkResponse.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionRequest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionResponse.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionRequest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionResponse.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
 3721113 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionLinkResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionVersionResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java
 99e4ccd 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ExtensionDAO.java 
PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ExtensionLinkDAO.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ExtensionEntity.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ExtensionLinkEntity.java
 PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/stack/BaseModule.java 
ef2438f 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 cbbdb91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ComponentModule.java 
65da145

Review Request 47957: Add support for Ubuntu 16

2016-05-27 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47957/
---

Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-16931
https://issues.apache.org/jira/browse/AMBARI-16931


Repository: ambari


Description
---

Microsoft needs support for Ubuntu 16 for HDInsight. The idea is that HDI 3.5
which corresponds to HDP 2.5 (Erie) will be on Ubuntu 16. We will only test
Ubuntu 16 on HDI, we won't test it for on-Prem. But, we do need Ambari to
support this OS.

So, there are two parts here. One is just to functionally make sure that
Ambari can accept doing an installation on Ubuntu. The second is to run Ambari
through its paces to make sure things actually work on Ubuntu 16.


Diffs
-

  ambari-common/src/main/python/ambari_commons/resources/os_family.json 92d9934 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/metainfo.xml
 781dee2 
  ambari-server/src/main/resources/common-services/GANGLIA/3.5.0/metainfo.xml 
f830967 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
16d0a84 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
7fc478f 
  ambari-server/src/main/resources/common-services/KAFKA/0.8.1/metainfo.xml 
c1dbf43 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 7d50ba1 
  ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/metainfo.xml 
7d8959d 
  ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/metainfo.xml 
adb5572 
  ambari-server/src/main/resources/common-services/RANGER/0.4.0/metainfo.xml 
9c33630 
  ambari-server/src/main/resources/common-services/SPARK/1.2.1/metainfo.xml 
4cd681a 
  ambari-server/src/main/resources/common-services/SPARK2/2.0.0/metainfo.xml 
98f1696 
  ambari-server/src/main/resources/common-services/STORM/0.9.3/metainfo.xml 
ed4337c 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml 
dee25ff 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml 
0b35357 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml 
1e1f8c3 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml 
437036f 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml 
2c40e25 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml 
22f93fe 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml 
6653670 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml 
95d6e2d 
  
ambari-server/src/main/resources/stacks/HDP/2.3.ECS/services/ZOOKEEPER/metainfo.xml
 4833857 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml 
57b19df 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml 
0abb468 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
875966c 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml 
57ee0b8 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml 
bb26f12 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml 
5e5c5cf 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml
 5804f88 
  ambari-server/src/main/resources/stacks/HDP/2.5/repos/repoinfo.xml e5c53ce 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml 
09791e4 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ZEPPELIN/metainfo.xml 
d4e9e3a 
  ambari-server/src/main/resources/version_definition.xsd 1c57e90 
  contrib/version-builder/version_builder.py 415e940 

Diff: https://reviews.apache.org/r/47957/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 47953: Add ability to query all hosts using hostname=% through AMS API.

2016-05-27 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47953/#review135240
---


Ship it!




Ship It!

- Aravindan Vijayan


On May 27, 2016, 12:47 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47953/
> ---
> 
> (Updated May 27, 2016, 12:47 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Myroslav Papirkovskyy, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-16926
> https://issues.apache.org/jira/browse/AMBARI-16926
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Useful for Top N feature
> Add ability to query all hosts using hostname=%.my.com through AMS API.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java
>  b1159c3 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestPhoenixTransactSQL.java
>  888234f 
> 
> Diff: https://reviews.apache.org/r/47953/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] ambari-metrics . SUCCESS [  1.041 
> s]
> [INFO] Ambari Metrics Common .. SUCCESS [  4.226 
> s]
> [INFO] Ambari Metrics Hadoop Sink . SUCCESS [  4.857 
> s]
> [INFO] Ambari Metrics Flume Sink .. SUCCESS [  2.434 
> s]
> [INFO] Ambari Metrics Kafka Sink .. SUCCESS [  3.550 
> s]
> [INFO] Ambari Metrics Storm Sink .. SUCCESS [  1.208 
> s]
> [INFO] Ambari Metrics Collector ... SUCCESS [04:05 
> min]
> [INFO] Ambari Metrics Monitor . SUCCESS [  1.147 
> s]
> [INFO] Ambari Metrics Grafana . SUCCESS [ 29.393 
> s]
> [INFO] Ambari Metrics Assembly  SUCCESS [02:47 
> min]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 47956: Automatically cleanup /var/run/ambari-server/stack-recommendations

2016-05-27 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47956/
---

(Updated May 27, 2016, 3:07 p.m.)


Review request for Ambari, Dmitro Lisnichenko and Srimanth Gunturi.


Bugs: AMBARI-16929
https://issues.apache.org/jira/browse/AMBARI-16929


Repository: ambari


Description
---

In clusters with many nodes and thousands of processes running
/var/run/ambari-server/stack-recommendations folder gets filled up and not
cleaned till ambari-server is restarted.

Disk is getting filled up causing space issues.


Diffs (updated)
-

  ambari-server/conf/unix/ambari.properties 9f1692e 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorHelper.java
 a925d7d 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ComponentLayoutRecommendationCommand.java
 0dff92b 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ComponentLayoutValidationCommand.java
 757ebee 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationDependenciesRecommendationCommand.java
 ae86548 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationRecommendationCommand.java
 ad01b40 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationValidationCommand.java
 c234947 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 3e20a09 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  ambari-server/src/main/java/org/apache/ambari/server/utils/DateUtils.java 
785f4fd 
  
ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationRecommendationCommandTest.java
 bc9cf77 
  
ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommandTest.java
 263bbe1 

Diff: https://reviews.apache.org/r/47956/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 47730: Improve TimelineMetricsCache eviction/flush logic using a cache library

2016-05-27 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47730/#review135239
---


Ship it!




Ship It!

- Aravindan Vijayan


On May 27, 2016, 2:28 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47730/
> ---
> 
> (Updated May 27, 2016, 2:28 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Myroslav Papirkovskyy, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-16821
> https://issues.apache.org/jira/browse/AMBARI-16821
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The TimelineMetricsCache implementation in the metrics sink side is currently 
> a ConcurrentSkipListMap. It is better to use pre built cache libraries like 
> Guava that offer more support.
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-common/pom.xml 70483c9 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java
>  0bed7d0 
>   
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java
>  18d973c 
>   
> ambari-metrics/ambari-metrics-hadoop-sink/src/test/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSinkTest.java
>  3d4a929 
> 
> Diff: https://reviews.apache.org/r/47730/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 47956: Automatically cleanup /var/run/ambari-server/stack-recommendations

2016-05-27 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47956/#review135238
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On May 27, 2016, 5:44 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47956/
> ---
> 
> (Updated May 27, 2016, 5:44 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Srimanth Gunturi.
> 
> 
> Bugs: AMBARI-16929
> https://issues.apache.org/jira/browse/AMBARI-16929
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In clusters with many nodes and thousands of processes running
> /var/run/ambari-server/stack-recommendations folder gets filled up and not
> cleaned till ambari-server is restarted.
> 
> Disk is getting filled up causing space issues.
> 
> 
> Diffs
> -
> 
>   ambari-server/conf/unix/ambari.properties 9f1692e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorHelper.java
>  a925d7d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ComponentLayoutRecommendationCommand.java
>  0dff92b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ComponentLayoutValidationCommand.java
>  757ebee 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationDependenciesRecommendationCommand.java
>  ae86548 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationRecommendationCommand.java
>  ad01b40 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationValidationCommand.java
>  c234947 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  3e20a09 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  7cfaf61 
>   ambari-server/src/main/java/org/apache/ambari/server/utils/DateUtils.java 
> 785f4fd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationRecommendationCommandTest.java
>  bc9cf77 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommandTest.java
>  263bbe1 
> 
> Diff: https://reviews.apache.org/r/47956/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 47956: Automatically cleanup /var/run/ambari-server/stack-recommendations

2016-05-27 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47956/
---

Review request for Ambari and Srimanth Gunturi.


Bugs: AMBARI-16929
https://issues.apache.org/jira/browse/AMBARI-16929


Repository: ambari


Description
---

In clusters with many nodes and thousands of processes running
/var/run/ambari-server/stack-recommendations folder gets filled up and not
cleaned till ambari-server is restarted.

Disk is getting filled up causing space issues.


Diffs
-

  ambari-server/conf/unix/ambari.properties 9f1692e 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorHelper.java
 a925d7d 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ComponentLayoutRecommendationCommand.java
 0dff92b 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ComponentLayoutValidationCommand.java
 757ebee 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationDependenciesRecommendationCommand.java
 ae86548 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationRecommendationCommand.java
 ad01b40 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationValidationCommand.java
 c234947 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 3e20a09 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 7cfaf61 
  ambari-server/src/main/java/org/apache/ambari/server/utils/DateUtils.java 
785f4fd 
  
ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/commands/ConfigurationRecommendationCommandTest.java
 bc9cf77 
  
ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommandTest.java
 263bbe1 

Diff: https://reviews.apache.org/r/47956/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 47730: Improve TimelineMetricsCache eviction/flush logic using a cache library

2016-05-27 Thread Dmytro Sen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47730/
---

(Updated Май 27, 2016, 2:28 п.п.)


Review request for Ambari, Aravindan Vijayan, Myroslav Papirkovskyy, and Sid 
Wagle.


Bugs: AMBARI-16821
https://issues.apache.org/jira/browse/AMBARI-16821


Repository: ambari


Description
---

The TimelineMetricsCache implementation in the metrics sink side is currently a 
ConcurrentSkipListMap. It is better to use pre built cache libraries like Guava 
that offer more support.


Diffs (updated)
-

  ambari-metrics/ambari-metrics-common/pom.xml 70483c9 
  
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java
 0bed7d0 
  
ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java
 18d973c 
  
ambari-metrics/ambari-metrics-hadoop-sink/src/test/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSinkTest.java
 3d4a929 

Diff: https://reviews.apache.org/r/47730/diff/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 47955: AMBARI-16906 Express upgrade: Oozie failed to start when user name and group are none-default values

2016-05-27 Thread Di Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47955/
---

(Updated May 27, 2016, 2:27 p.m.)


Review request for Ambari and Alejandro Fernandez.


Summary (updated)
-

AMBARI-16906 Express upgrade: Oozie failed to start when user name and group 
are none-default values


Bugs: AMBARI-16906
https://issues.apache.org/jira/browse/AMBARI-16906


Repository: ambari


Description
---

oozie_server_upgrade.py has hardcoded user name to oozie and group to hadoop. 
Both values should have been pulled from params.py


Diffs
-

  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
 a1f3336 

Diff: https://reviews.apache.org/r/47955/diff/


Testing
---

patch a trunk cluster (Oozie use is non-default myoozie, group is non-default 
myhadoop) with the change, run express upgrade, verify that with the change, 
oozie restarted successfully during the EU.


Thanks,

Di Li

Review Request 47955: AmbariAMBARI-16906 Express upgrade: Oozie failed to start when user name and group are none-default values

2016-05-27 Thread Di Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47955/
---

Review request for Ambari and Alejandro Fernandez.


Bugs: AMBARI-16906
https://issues.apache.org/jira/browse/AMBARI-16906


Repository: ambari


Description
---

oozie_server_upgrade.py has hardcoded user name to oozie and group to hadoop. 
Both values should have been pulled from params.py


Diffs
-

  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
 a1f3336 

Diff: https://reviews.apache.org/r/47955/diff/


Testing
---

patch a trunk cluster (Oozie use is non-default myoozie, group is non-default 
myhadoop) with the change, run express upgrade, verify that with the change, 
oozie restarted successfully during the EU.


Thanks,

Di Li

Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services

2016-05-27 Thread Tim Thorpe


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47656/
---

(Updated May 27, 2016, 2:25 p.m.)


Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh 
chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, 
and Yusaku Sako.


Changes
---

Cleaned up commented out code and fixed a couple of issues with the extension 
REST APIs (things that were missed during a previous merge).


Bugs: AMBARI-12885
https://issues.apache.org/jira/browse/AMBARI-12885


Repository: ambari


Description
---

The purpose of this proposal is to facilitate adding custom services to an 
existing stack. Ideally this would support adding and upgrading custom services 
separately from the core services defined in the stack. In particular we are 
looking at custom services that need to support several different stacks 
(different distributions of Ambari). The release cycle of the custom services 
may be different from that of the core stack; that is, a custom service may be 
upgraded at a different rate than the core distribution itself and may be 
upgraded multiple times within the lifespan of a single release of the core 
distribution.

One possible approach to handling this would be dynamically extending a stack 
(after install time). It would be best to extend the stack in packages where a 
stack extension package can have one or more custom services.


Diffs (updated)
-

  ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 
fcea23f 
  ambari-agent/src/main/python/ambari_agent/FileCache.py b7c5dee 
  ambari-server/conf/unix/ambari.properties 9f1692e 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
 cf7c391 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 c54fe3f 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java
 557ce98 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 f5e8f39 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 d6b9d0e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 dd3d098 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkRequest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkResponse.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionRequest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionResponse.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionRequest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionResponse.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
 3721113 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionLinkResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionVersionResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java
 99e4ccd 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ExtensionDAO.java 
PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ExtensionLinkDAO.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ExtensionEntity.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ExtensionLinkEntity.java
 PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/stack/BaseModule.java 
ef2438f 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 cbbdb91

Review Request 47954: AMBARI-16907 Ambari web UI does not auto-set all required properties when user toggles on the Yarn CPU isolation feature on the web UI

2016-05-27 Thread Di Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47954/
---

Review request for Ambari and Alejandro Fernandez.


Bugs: AMBARI-16907
https://issues.apache.org/jira/browse/AMBARI-16907


Repository: ambari


Description
---

The document here mentioned five properties that need to be updated when the 
CGroups with Yarn feature is enabled via Ambari web UI. 
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html

Right now, however, only one is auto-changed – 
"yarn.nodemanager.container-executor.class".

This is due to a mix of yarn-site.xml metainfo change and the properties name 
used in the stack_advisor.py


Diffs
-

  
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml
 905383a 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
03e8150 

Diff: https://reviews.apache.org/r/47954/diff/


Testing
---

manually patch a trunk cluster with changes and tested that with the changes, 
all CGroup required properties are changed when I enabled CPU isolation on the 
Yarn config tab.


Thanks,

Di Li

Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services

2016-05-27 Thread Tim Thorpe



> On May 23, 2016, 10:49 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java,
> >  line 305
> > 
> >
> > a lot of commented out code here

I have removed most if not all of the commented code.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47656/#review134330
---


On May 20, 2016, 7:03 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47656/
> ---
> 
> (Updated May 20, 2016, 7:03 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh 
> chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth 
> Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-12885
> https://issues.apache.org/jira/browse/AMBARI-12885
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The purpose of this proposal is to facilitate adding custom services to an 
> existing stack. Ideally this would support adding and upgrading custom 
> services separately from the core services defined in the stack. In 
> particular we are looking at custom services that need to support several 
> different stacks (different distributions of Ambari). The release cycle of 
> the custom services may be different from that of the core stack; that is, a 
> custom service may be upgraded at a different rate than the core distribution 
> itself and may be upgraded multiple times within the lifespan of a single 
> release of the core distribution.
> 
> One possible approach to handling this would be dynamically extending a stack 
> (after install time). It would be best to extend the stack in packages where 
> a stack extension package can have one or more custom services.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 
> fcea23f 
>   ambari-agent/src/main/python/ambari_agent/FileCache.py b7c5dee 
>   ambari-server/conf/unix/ambari.properties 9f1692e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
>  9c864b6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  c54fe3f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java
>  557ce98 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  f5e8f39 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
>  d6b9d0e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  dd3d098 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkRequest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkResponse.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionRequest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionResponse.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionRequest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionResponse.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
>  3721113 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionLinkResourceProvider.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionResourceProvider.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionVersionResourceProvider.java
>  PRE-CREATION 
>   
>

Re: Review Request 47783: Cluster operator and cluster admin not allowed to install ambari agent

2016-05-27 Thread Robert Levas



> On May 24, 2016, 4:41 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java,
> >  lines 190-194
> > 
> >
> > Should special permissions like this go right in the action definition 
> > itself?  Would require finding out if the file is readable by non-root 
> > Ambari.  Would help with having to hard code action names here.
> 
> Robert Levas wrote:
> I dont think I understand the issue.  
> 
> The request to create a Request resource with the command "check_host" 
> needs to be processed to ensure that the user requesting this operation is 
> authorized to do so.  This check cannot be done anywhere else since we dont 
> know until this point what the user is trying to do - that is without parsing 
> the request data an additional time just for the authorization check.
> 
> Nate Cole wrote:
> I really only meant that check_host, and all other custom actions, are 
> defined in a-s/src/main/resources/custom_action_definitions.  Should the 
> permissions needed to run be set with the definition, not special-cased in 
> code?

Nate, have a look at the new patch.. I think this is what you were looking for.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47783/#review134630
---


On May 27, 2016, 9:52 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47783/
> ---
> 
> (Updated May 27, 2016, 9:52 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-16851
> https://issues.apache.org/jira/browse/AMBARI-16851
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Cluster operator and the cluster admin must be allowed to add/delete hosts 
> but install of agents using /bootstrap fails with 403
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  5b318af 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinition.java
>  1f189a3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionManager.java
>  97aa8a0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionSpec.java
>  2dfa1f2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariAuthorizationFilter.java
>  5c74f07 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  0deba5d 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 2c2d743 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql ee87cc5 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql edc46f7 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 6f38ec8 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ca57de5 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 61aadf0 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 9269b13 
>   
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml
>  6304baf 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  3ec9cb3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ActionResourceProviderTest.java
>  96995b4 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  65efc63 
>   
> ambari-server/src/test/java/org/apache/ambari/server/customactions/ActionDefinitionManagerTest.java
>  ec84922 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/TestAuthenticationFactory.java
>  69b4b08 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  4dedc98 
>   
> ambari-server/src/test/resources/custom_action_definitions/cust_action_definitions1.xml
>  9cee575 
>   
> ambari-server/src/test/resources/custom_action_definitions_invalid/cust_action_definitions_invalid.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47783/diff/
> 
> 
> Testing
> ---
> 
> Manually tested, newly created cluster and upgrade 
> 
> # Local test results:
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:15:22.164s
> [INFO] Finished at: Tue May 24 13:28:21 EDT 2016
> [INFO] Final Memory: 59M/1807M

Re: Review Request 47783: Cluster operator and cluster admin not allowed to install ambari agent

2016-05-27 Thread Robert Levas



> On May 25, 2016, 4:21 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java,
> >  lines 190-194
> > 
> >
> > This is a little odd - string compares? We don't have anything more 
> > contractual for "check host" permissions?
> 
> Robert Levas wrote:
> Not really. Essentually the (custom) commands sent to the Reqeust 
> endpoint are strings that (I think) get interpreted on the agent side.  As 
> @ncole pointed out, there is a file at 
> `custom_action_definitions/system_action_definitions.xml` that defines these 
> (or at least some of them) but the definitions are lacking.

Jonathan... have a look at the new patch... see if this is a better solution.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47783/#review134823
---


On May 27, 2016, 9:52 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47783/
> ---
> 
> (Updated May 27, 2016, 9:52 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Myroslav Papirkovskyy, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-16851
> https://issues.apache.org/jira/browse/AMBARI-16851
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Cluster operator and the cluster admin must be allowed to add/delete hosts 
> but install of agents using /bootstrap fails with 403
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  5b318af 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinition.java
>  1f189a3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionManager.java
>  97aa8a0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/customactions/ActionDefinitionSpec.java
>  2dfa1f2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariAuthorizationFilter.java
>  5c74f07 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  0deba5d 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 2c2d743 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql ee87cc5 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql edc46f7 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 6f38ec8 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ca57de5 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 61aadf0 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 9269b13 
>   
> ambari-server/src/main/resources/custom_action_definitions/system_action_definitions.xml
>  6304baf 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  3ec9cb3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ActionResourceProviderTest.java
>  96995b4 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  65efc63 
>   
> ambari-server/src/test/java/org/apache/ambari/server/customactions/ActionDefinitionManagerTest.java
>  ec84922 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/TestAuthenticationFactory.java
>  69b4b08 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  4dedc98 
>   
> ambari-server/src/test/resources/custom_action_definitions/cust_action_definitions1.xml
>  9cee575 
>   
> ambari-server/src/test/resources/custom_action_definitions_invalid/cust_action_definitions_invalid.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47783/diff/
> 
> 
> Testing
> ---
> 
> Manually tested, newly created cluster and upgrade 
> 
> # Local test results:
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:15:22.164s
> [INFO] Finished at: Tue May 24 13:28:21 EDT 2016
> [INFO] Final Memory: 59M/1807M
> [INFO] 
> 
> 
> #Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 47953: Add ability to query all hosts using hostname=% through AMS API.

2016-05-27 Thread Dmytro Sen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47953/
---

(Updated Май 27, 2016, 12:47 п.п.)


Review request for Ambari, Aravindan Vijayan, Myroslav Papirkovskyy, and Sid 
Wagle.


Bugs: AMBARI-16926
https://issues.apache.org/jira/browse/AMBARI-16926


Repository: ambari


Description
---

Useful for Top N feature
Add ability to query all hosts using hostname=%.my.com through AMS API.


Diffs
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java
 b1159c3 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestPhoenixTransactSQL.java
 888234f 

Diff: https://reviews.apache.org/r/47953/diff/


Testing (updated)
---

Unit tests passed

[INFO] Reactor Summary:
[INFO] 
[INFO] ambari-metrics . SUCCESS [  1.041 s]
[INFO] Ambari Metrics Common .. SUCCESS [  4.226 s]
[INFO] Ambari Metrics Hadoop Sink . SUCCESS [  4.857 s]
[INFO] Ambari Metrics Flume Sink .. SUCCESS [  2.434 s]
[INFO] Ambari Metrics Kafka Sink .. SUCCESS [  3.550 s]
[INFO] Ambari Metrics Storm Sink .. SUCCESS [  1.208 s]
[INFO] Ambari Metrics Collector ... SUCCESS [04:05 min]
[INFO] Ambari Metrics Monitor . SUCCESS [  1.147 s]
[INFO] Ambari Metrics Grafana . SUCCESS [ 29.393 s]
[INFO] Ambari Metrics Assembly  SUCCESS [02:47 min]
[INFO] 
[INFO] BUILD SUCCESS


Thanks,

Dmytro Sen

1 2 >

1 - 100 of 117 matches

Mail list logo