Review Request 51837: Make Execute timeout to be able to kill process trees which doesn't respond to SIGTERM

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51837/
---

Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-18369
https://issues.apache.org/jira/browse/AMBARI-18369


Repository: ambari


Description
---


Diffs
-

  ambari-common/src/main/python/resource_management/core/__init__.py 1af793b 
  ambari-common/src/main/python/resource_management/core/files/killtree.sh 
PRE-CREATION 
  ambari-common/src/main/python/resource_management/core/providers/system.py 
fcbab01 
  ambari-common/src/main/python/resource_management/core/resources/system.py 
7f164f6 
  ambari-common/src/main/python/resource_management/core/shell.py 6d9eb18 
  ambari-common/src/main/python/resource_management/core/signal_utils.py 
PRE-CREATION 
  ambari-common/src/main/python/resource_management/core/utils.py dc771d1 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py
 e02ed5a 

Diff: https://reviews.apache.org/r/51837/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 51837: Make Execute timeout to be able to kill process trees which doesn't respond to SIGTERM

[Dmitro Lisnichenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51837/#review148668
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On Sept. 13, 2016, 10:19 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51837/
> ---
> 
> (Updated Sept. 13, 2016, 10:19 a.m.)
> 
> 
> Review request for Ambari and Dmitro Lisnichenko.
> 
> 
> Bugs: AMBARI-18369
> https://issues.apache.org/jira/browse/AMBARI-18369
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/resource_management/core/__init__.py 1af793b 
>   ambari-common/src/main/python/resource_management/core/files/killtree.sh 
> PRE-CREATION 
>   ambari-common/src/main/python/resource_management/core/providers/system.py 
> fcbab01 
>   ambari-common/src/main/python/resource_management/core/resources/system.py 
> 7f164f6 
>   ambari-common/src/main/python/resource_management/core/shell.py 6d9eb18 
>   ambari-common/src/main/python/resource_management/core/signal_utils.py 
> PRE-CREATION 
>   ambari-common/src/main/python/resource_management/core/utils.py dc771d1 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py
>  e02ed5a 
> 
> Diff: https://reviews.apache.org/r/51837/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 51838: Do not execute stack-selector-tool after HDP component install on sysprepped hosts.

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51838/
---

Review request for Ambari, Andrew Onischuk, Jayush Luniya, and Sumit Mohanty.


Bugs: AMBARI-18370
https://issues.apache.org/jira/browse/AMBARI-18370


Repository: ambari


Description
---

1. Modified Ambaripreupload.py to invoke 'hdp-select set all' to ensure that 
when preparing a sysprepped host the stack selector tool was executed for the 
selected HDP components
2. The shared initialisation script that is executed once after each HDP 
component install as AFTER-install hook skips executing 'hdp-select set all' if 
it's a sysprepped host


Diffs
-

  ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
 819d8f7 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
 9982dc6 
  
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
 06a366e 

Diff: https://reviews.apache.org/r/51838/diff/


Testing
---

Manual testing.

Python unit tests:
--
Total run:1125
Total errors:0
Total failures:0
OK
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:43.849s
[INFO] Finished at: Tue Sep 13 09:56:49 CEST 2016
[INFO] Final Memory: 73M/799M
[INFO] 


Thanks,

Sebastian Toader

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/
---

(Updated Sept. 13, 2016, 4:59 a.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.


Bugs: AMBARI-18365
https://issues.apache.org/jira/browse/AMBARI-18365


Repository: ambari


Description
---

Add the followng Ambari configuration options to support Kerberos token 
authentication

- `authentication.kerberos.enabled`
-- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
Ambari:  {{true}} to enable this feature; {{false}}, otherwise
- `authentication.kerberos.spnego.principal`
-- The Kerberos principal name to use when verifying user-supplied Kerberos 
tokens for authentication via SPNEGO
- `authentication.kerberos.spnego.keytab.file`
-- The Kerberos keytab file to use when verifying user-supplied Kerberos tokens 
for authentication via SPNEGO
- `authentication.kerberos.user.types`
-- A comma-delimited (ordered) list of preferred user types to use when finding 
the Ambari user account for the user-supplied Kerberos identity during 
authentication via SPNEGO
- `authentication.kerberos.auth_to_local.rules`
-- The auth-to-local rules set to use when translating a user's principal name 
to a local user name during authentication via SPNEGO.

NOTE: These properties are in the {{ambari.properties}} file since this feature 
may be enabled whether the rest of the cluster has Kerberos enabled or not. 

See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
requiring the need for this patch.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 ee73b8d 
  
ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
 f429a36 
  
ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/51822/diff/


Testing (updated)
---

Manually tested...

# Local test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 2:51:23.081s
[INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
[INFO] Final Memory: 48M/1792M
[INFO] 

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/
---

(Updated Sept. 13, 2016, 5 a.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.


Bugs: AMBARI-18365
https://issues.apache.org/jira/browse/AMBARI-18365


Repository: ambari


Description
---

Add the followng Ambari configuration options to support Kerberos token 
authentication

- `authentication.kerberos.enabled`
-- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
Ambari:  {{true}} to enable this feature; {{false}}, otherwise
- `authentication.kerberos.spnego.principal`
-- The Kerberos principal name to use when verifying user-supplied Kerberos 
tokens for authentication via SPNEGO
- `authentication.kerberos.spnego.keytab.file`
-- The Kerberos keytab file to use when verifying user-supplied Kerberos tokens 
for authentication via SPNEGO
- `authentication.kerberos.user.types`
-- A comma-delimited (ordered) list of preferred user types to use when finding 
the Ambari user account for the user-supplied Kerberos identity during 
authentication via SPNEGO
- `authentication.kerberos.auth_to_local.rules`
-- The auth-to-local rules set to use when translating a user's principal name 
to a local user name during authentication via SPNEGO.

NOTE: These properties are in the {{ambari.properties}} file since this feature 
may be enabled whether the rest of the cluster has Kerberos enabled or not. 

See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
requiring the need for this patch.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 ee73b8d 
  
ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
 f429a36 
  
ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/51822/diff/


Testing (updated)
---

Manually tested...

# Local test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 2:51:23.081s
[INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
[INFO] Final Memory: 48M/1792M
[INFO] 

# Jenkins test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 01:34 h
[INFO] Finished at: 2016-09-13T08:46:23+00:00
[INFO] Final Memory: 178M/2672M
[INFO] 

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12828101/AMBARI-18365_trunk_01.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
ambari-server.

Test results: 
https://builds.apache.org/job/Ambari-trunk-test-patch/8643//testReport/
Console output: 
https://builds.apache.org/job/Ambari-trunk-test-patch/8643//console

This message is automatically generated.


Thanks,

Robert Levas

Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services

[Balázs Bence Sári]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51238/
---

(Updated Sept. 13, 2016, 9:25 a.m.)


Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and 
Sebastian Toader.


Changes
---

Merged trunk and fixed stylistic issues.


Bugs: AMBARI-15538
https://issues.apache.org/jira/browse/AMBARI-15538


Repository: ambari


Description
---

Implementation contains the following things:
- On startup, add-on service repos are loaded into the stack model
- On startup, add-on service repos are merged into the VDF's downloaded by 
LatestRepoCallable
- On startup, if the is an existing cluster, it's repository version entity is 
potentially updated with new add-on service repos.
- Repository definitions contain two new optional fields: service_name and 
service_version. (null for stack repositories) 
- Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5)
- New unit tests


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
 7403dba 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 1d82928 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 bae4ab6 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
 02fc2ec 
  ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java 
PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
bc94104 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java 
3acc617 
  ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
1fee589 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 7bcd08b 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java 
83a 
  ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
14ff9de 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
 d26b834 
  ambari-server/src/main/resources/version_definition.xsd bd49028 
  ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java 
PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java
 1d73ff3 
  
ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java 
PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java
 PRE-CREATION 
  
ambari-server/src/test/resources/common-services/ADDON/1.0/configuration/addon-env.xml
 PRE-CREATION 
  ambari-server/src/test/resources/common-services/ADDON/1.0/metainfo.xml 
PRE-CREATION 
  
ambari-server/src/test/resources/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest_initialRepos.json
 PRE-CREATION 
  
ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/metainfo.xml
 PRE-CREATION 
  
ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/repos/repoinfo.xml
 PRE-CREATION 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/configuration/microsoft-r-env.xml
 PRE-CREATION 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/microsoft_r.py
 61ea96b 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/custom-services/MICROSOFT_R/8.0.0/repos/repoinfo.xml
 PRE-CREATION 

Diff: https://reviews.apache.org/r/51238/diff/


Testing
---

- Manually tested
- Wrote new unit tests
- All unit tests passed except two which were failing in CI builds as well.


Thanks,

Balázs Bence Sári

Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services

[Balázs Bence Sári]

> On Sept. 7, 2016, 6:35 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java,
> >  lines 166-169
> > 
> >
> > Objects.equals() for simplicity.

Fixed.


> On Sept. 7, 2016, 6:35 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java,
> >  lines 179-181
> > 
> >
> > Use Objects.hash(...) for easier going

Fixed.


- Balázs Bence


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51238/#review148053
---


On Sept. 13, 2016, 9:25 a.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51238/
> ---
> 
> (Updated Sept. 13, 2016, 9:25 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-15538
> https://issues.apache.org/jira/browse/AMBARI-15538
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Implementation contains the following things:
> - On startup, add-on service repos are loaded into the stack model
> - On startup, add-on service repos are merged into the VDF's downloaded by 
> LatestRepoCallable
> - On startup, if the is an existing cluster, it's repository version entity 
> is potentially updated with new add-on service repos.
> - Repository definitions contain two new optional fields: service_name and 
> service_version. (null for stack repositories) 
> - Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5)
> - New unit tests
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
>  7403dba 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  1d82928 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  bae4ab6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
>  02fc2ec 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> bc94104 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java
>  3acc617 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 1fee589 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  7bcd08b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java
>  83a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> 14ff9de 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
>  d26b834 
>   ambari-server/src/main/resources/version_definition.xsd bd49028 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java
>  1d73ff3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/common-services/ADDON/1.0/configuration/addon-env.xml
>  PRE-CREATION 
>   ambari-server/src/test/resources/common-services/ADDON/1.0/metainfo.xml 
> PRE-CREATION 
>   
> ambari-server/src/test/resources/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest_initialRepos.json
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/repos/repoinfo.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/configuration/microsoft-r-env.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/microsoft_r.py
>  61ea96b 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/custom-services/MICROSOFT_R/8.0.0/repos/repoinfo.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51238/diff/
> 
> 
> Testing
> ---

Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services

[Balázs Bence Sári]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51238/
---

(Updated Sept. 13, 2016, 9:31 a.m.)


Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and 
Sebastian Toader.


Changes
---

Minor formatting fixes


Bugs: AMBARI-15538
https://issues.apache.org/jira/browse/AMBARI-15538


Repository: ambari


Description
---

Implementation contains the following things:
- On startup, add-on service repos are loaded into the stack model
- On startup, add-on service repos are merged into the VDF's downloaded by 
LatestRepoCallable
- On startup, if the is an existing cluster, it's repository version entity is 
potentially updated with new add-on service repos.
- Repository definitions contain two new optional fields: service_name and 
service_version. (null for stack repositories) 
- Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5)
- New unit tests


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
 7403dba 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 1d82928 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 bae4ab6 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
 02fc2ec 
  ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java 
PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
bc94104 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java 
3acc617 
  ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
1fee589 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 7bcd08b 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java 
83a 
  ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
14ff9de 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
 d26b834 
  ambari-server/src/main/resources/version_definition.xsd bd49028 
  ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java 
PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java
 1d73ff3 
  
ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java 
PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java
 PRE-CREATION 
  
ambari-server/src/test/resources/common-services/ADDON/1.0/configuration/addon-env.xml
 PRE-CREATION 
  ambari-server/src/test/resources/common-services/ADDON/1.0/metainfo.xml 
PRE-CREATION 
  
ambari-server/src/test/resources/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest_initialRepos.json
 PRE-CREATION 
  
ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/metainfo.xml
 PRE-CREATION 
  
ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/repos/repoinfo.xml
 PRE-CREATION 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/configuration/microsoft-r-env.xml
 PRE-CREATION 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/microsoft_r.py
 61ea96b 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/custom-services/MICROSOFT_R/8.0.0/repos/repoinfo.xml
 PRE-CREATION 

Diff: https://reviews.apache.org/r/51238/diff/


Testing
---

- Manually tested
- Wrote new unit tests
- All unit tests passed except two which were failing in CI builds as well.


Thanks,

Balázs Bence Sári

Re: Review Request 51838: Do not execute stack-selector-tool after HDP component install on sysprepped hosts.

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51838/#review148686
---


Ship it!




Ship It!

- Andrew Onischuk


On Sept. 13, 2016, 7:56 a.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51838/
> ---
> 
> (Updated Sept. 13, 2016, 7:56 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jayush Luniya, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18370
> https://issues.apache.org/jira/browse/AMBARI-18370
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1. Modified Ambaripreupload.py to invoke 'hdp-select set all' to ensure that 
> when preparing a sysprepped host the stack selector tool was executed for the 
> selected HDP components
> 2. The shared initialisation script that is executed once after each HDP 
> component install as AFTER-install hook skips executing 'hdp-select set all' 
> if it's a sysprepped host
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  819d8f7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  9982dc6 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
>  06a366e 
> 
> Diff: https://reviews.apache.org/r/51838/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Python unit tests:
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:43.849s
> [INFO] Finished at: Tue Sep 13 09:56:49 CEST 2016
> [INFO] Final Memory: 73M/799M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services

[Balázs Bence Sári]

> On Sept. 7, 2016, 6:35 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java,
> >  lines 42-43
> > 
> >
> > Syntax.

Fixed.


> On Sept. 7, 2016, 6:35 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java,
> >  lines 48-49
> > 
> >
> > Odd syntax.  Usually see:
> > 
> > @Nullable
> > private String repoDir;

Fixed.


- Balázs Bence


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51238/#review148053
---


On Sept. 13, 2016, 9:31 a.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51238/
> ---
> 
> (Updated Sept. 13, 2016, 9:31 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-15538
> https://issues.apache.org/jira/browse/AMBARI-15538
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Implementation contains the following things:
> - On startup, add-on service repos are loaded into the stack model
> - On startup, add-on service repos are merged into the VDF's downloaded by 
> LatestRepoCallable
> - On startup, if the is an existing cluster, it's repository version entity 
> is potentially updated with new add-on service repos.
> - Repository definitions contain two new optional fields: service_name and 
> service_version. (null for stack repositories) 
> - Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5)
> - New unit tests
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
>  7403dba 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  1d82928 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  bae4ab6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
>  02fc2ec 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> bc94104 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java
>  3acc617 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 1fee589 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  7bcd08b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java
>  83a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> 14ff9de 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
>  d26b834 
>   ambari-server/src/main/resources/version_definition.xsd bd49028 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java
>  1d73ff3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/common-services/ADDON/1.0/configuration/addon-env.xml
>  PRE-CREATION 
>   ambari-server/src/test/resources/common-services/ADDON/1.0/metainfo.xml 
> PRE-CREATION 
>   
> ambari-server/src/test/resources/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest_initialRepos.json
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/repos/repoinfo.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/configuration/microsoft-r-env.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/microsoft_r.py
>  61ea96b 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/custom-services/MICROSOFT_R/8.0.0/repos/repoinfo.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51238/dif

Re: Review Request 49804: improve resourcemanager HA description

[wang yaoxin]

> On 九月 12, 2016, 9:08 a.m., Aleksandr Kovalenko wrote:
> > ambari-web/app/models/host_component.js, line 333
> > 
> >
> > What is the reason for this change?
> > Why we decide to not show Enable label if HA is already enabled?

Thank you ,Aleksandr . Just like the NameNode HA , it's better to decide the RM 
Enable lable and cssClass


- wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49804/#review148468
---


On 九月 12, 2016, 3:24 a.m., wang yaoxin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49804/
> ---
> 
> (Updated 九月 12, 2016, 3:24 a.m.)
> 
> 
> Review request for Ambari, Aleksandr Kovalenko, jun aoki, Zhe (Joe) Wang, 
> Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17518
> https://issues.apache.org/jira/browse/AMBARI-17518
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> improve the resourcemanager HA description
> 
> 
> Diffs
> -
> 
>   ambari-web/app/models/host_component.js 973b809 
> 
> Diff: https://reviews.apache.org/r/49804/diff/
> 
> 
> Testing
> ---
> 
> done
> 
> 
> Thanks,
> 
> wang yaoxin
> 
>

Re: Review Request 51811: All classes recompiled due to Maven bug, even if none changed

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51811/#review148700
---


Ship it!




Ship It!

- Nate Cole


On Sept. 12, 2016, 11:56 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51811/
> ---
> 
> (Updated Sept. 12, 2016, 11:56 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Laszlo Puskas, Nate Cole, Oliver 
> Szabo, Sebastian Toader, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-18361
> https://issues.apache.org/jira/browse/AMBARI-18361
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix:
> 
> 1. Upgrade to maven-compiler-plugin from 3.0 (because 
> https://issues.apache.org/jira/browse/MCOMPILER-187 was only fixed in 3.1+) 
> to 3.2 (because it is already used by ambari-logsearch and ambari-metrics)
> 
> 2. Set useIncrementalCompilation flag to false (defaults to true) to work 
> around https://issues.apache.org/jira/browse/MCOMPILER-209
> 
> 
> Diffs
> -
> 
>   ambari-server/pom.xml 45d3ce4 
> 
> Diff: https://reviews.apache.org/r/51811/diff/
> 
> 
> Testing
> ---
> 
> 1. Verify Maven reports "Nothing to compile - all classes are up to date" if 
> no class was changed.
> 
> ```
> mvn -pl ambari-server compile
> mvn -pl ambari-server compile
> ```
> 
> 2. Verify Maven only reports compilation of 3 source files after 3 source 
> files updated.
> 
> ```
> mvn -pl ambari-server compile
> find ambari-server/src/main/java -type f | head -3 | xargs touch
> mvn -pl ambari-server compile
> ```
> 
> 3. Verify Maven compiles all Java sources after removing class files.
> 
> ```
> mvn -pl ambari-server compile
> find ambari-server -name '*.class' | xargs rm
> mvn -pl ambari-server compile
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148699
---




ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (line 2376)


convenience



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (line 2585)


authentication



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (line 2586)


Although I understand the motivation, I still don't like doing this since 
getting the backing Properties object for the entire Configuration would be 
missing these properties. Can we mirror them in the normal Properties object as 
well?



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (lines 5292 - 5294)


Why not just return here which would prevent you from needing to embed the 
rest of the logic in the if-statement



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (lines 5298 - 5301)


KERBEROS_AUTH_USER_TYPES has a default value, so `getProperty` will never 
return null.



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (lines 5310 - 5311)


Invalid types would seem to be a mis-configuration which could cause 
connectivity issues with Ambari; did you want to instead throw a real exception 
here?



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (lines 5318 - 5322)


Not sure if this is possible since KERBEROS_AUTH_USER_TYPES has a default 
value.



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (lines 5345 - 5348)


Same question as above - because these configurations involve connection to 
Ambari, should we instead throw exceptions and fail startup on problems like 
this?


- Jonathan Hurley


On Sept. 13, 2016, 5 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 5 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> --

Review Request 51843: rolling restart datanode cluster name in audit log show null

[wang yaoxin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51843/
---

Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sebastian 
Toader.


Bugs: AMBARI-18374
https://issues.apache.org/jira/browse/AMBARI-18374


Repository: ambari


Description
---

when rolling restart services ( eg:DataNode ) ,cluster name in audit log show 
null

2016-09-13T17:29:22.033+0800, User(admin), RemoteIp(127.0.0.1), 
Operation(Request from server), RequestType(POST), 
url(http://localhost:8080/api/v1/clusters/amabri/requests), ResultStatus(202 
Accepted), Command(RESTART), Cluster name(null)

2016-09-13T17:29:22.042+0800, User(admin), 
Operation(PARSE.ROLLING-RESTART.DATANODE.1.1), Status(IN_PROGRESS), 
RequestId(15)


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/audit/request/eventcreator/RequestEventCreator.java
 075e328 

Diff: https://reviews.apache.org/r/51843/diff/


Testing
---

done!


Thanks,

wang yaoxin

Re: Review Request 51811: All classes recompiled due to Maven bug, even if none changed

[Laszlo Puskas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51811/#review148702
---


Ship it!




Ship It!

- Laszlo Puskas


On Sept. 12, 2016, 3:56 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51811/
> ---
> 
> (Updated Sept. 12, 2016, 3:56 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Laszlo Puskas, Nate Cole, Oliver 
> Szabo, Sebastian Toader, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-18361
> https://issues.apache.org/jira/browse/AMBARI-18361
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix:
> 
> 1. Upgrade to maven-compiler-plugin from 3.0 (because 
> https://issues.apache.org/jira/browse/MCOMPILER-187 was only fixed in 3.1+) 
> to 3.2 (because it is already used by ambari-logsearch and ambari-metrics)
> 
> 2. Set useIncrementalCompilation flag to false (defaults to true) to work 
> around https://issues.apache.org/jira/browse/MCOMPILER-209
> 
> 
> Diffs
> -
> 
>   ambari-server/pom.xml 45d3ce4 
> 
> Diff: https://reviews.apache.org/r/51811/diff/
> 
> 
> Testing
> ---
> 
> 1. Verify Maven reports "Nothing to compile - all classes are up to date" if 
> no class was changed.
> 
> ```
> mvn -pl ambari-server compile
> mvn -pl ambari-server compile
> ```
> 
> 2. Verify Maven only reports compilation of 3 source files after 3 source 
> files updated.
> 
> ```
> mvn -pl ambari-server compile
> find ambari-server/src/main/java -type f | head -3 | xargs touch
> mvn -pl ambari-server compile
> ```
> 
> 3. Verify Maven compiles all Java sources after removing class files.
> 
> ```
> mvn -pl ambari-server compile
> find ambari-server -name '*.class' | xargs rm
> mvn -pl ambari-server compile
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148701
---


Ship it!




LGTM, just confirming that if these are properties specific for SPNEGO, then I 
think we should say that for property names.


ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (lines 1334 - 1336)


Is this specifically for SPNEGO, or any Kerberos auth?  If so, then the 
property should use the word "spnego" somewhere like you have for other 
properties.


- Nate Cole


On Sept. 13, 2016, 5 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 5 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:51:23.081s
> [INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
> [INFO] Final Memory: 48M/1792M
> [INFO] 
> 
> 
> # Jenkins test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 01:34 h
> [INFO] Finished at: 2016-09-13T08:46:23+00:00
> [INFO] Final Memory: 178M/2672M
> [INFO] 
> 
> 
> {color:green}+1 overall{color}.  Here are the results of testing the latest 
> attachment 
>   
> http://issues.apache.org/jira/secure/attachment/12828101/AMBARI-18365_trunk_01.patch
>   against trunk revision .
> 
> {color:green}+1 @author{color}.  The patch does not contain any @author 
> tags.
> 
> {color:green}+1 tests included{color}.  The patch appears to include 2 
> new or modified test files.
> 
> {color:green}+1 javac{color}.  The applied patch does not increase the 
> total number of javac compiler warnings.
> 
> {color:green}+1 release audit{color}.  The applied patch does not 
> increase the total number of release audit warnings.
> 
> {color:green}+1 core tests{color}.  The patch passed unit tests in 
> ambari-server.
> 
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/8643//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch

Re: Review Request 51843: rolling restart datanode cluster name in audit log show null

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51843/#review148703
---




ambari-server/src/main/java/org/apache/ambari/server/audit/request/eventcreator/RequestEventCreator.java
 (line 90)


Please add a unit test for this change.


- Sebastian Toader


On Sept. 13, 2016, 2:25 p.m., wang yaoxin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51843/
> ---
> 
> (Updated Sept. 13, 2016, 2:25 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-18374
> https://issues.apache.org/jira/browse/AMBARI-18374
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> when rolling restart services ( eg:DataNode ) ,cluster name in audit log show 
> null
> 
> 2016-09-13T17:29:22.033+0800, User(admin), RemoteIp(127.0.0.1), 
> Operation(Request from server), RequestType(POST), 
> url(http://localhost:8080/api/v1/clusters/amabri/requests), ResultStatus(202 
> Accepted), Command(RESTART), Cluster name(null)
> 
> 2016-09-13T17:29:22.042+0800, User(admin), 
> Operation(PARSE.ROLLING-RESTART.DATANODE.1.1), Status(IN_PROGRESS), 
> RequestId(15)
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/request/eventcreator/RequestEventCreator.java
>  075e328 
> 
> Diff: https://reviews.apache.org/r/51843/diff/
> 
> 
> Testing
> ---
> 
> done!
> 
> 
> Thanks,
> 
> wang yaoxin
> 
>

Re: Review Request 51705: Password in the configurations.json file in the ambari-agent cache is not encrypted

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51705/#review148705
---




ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py (lines 137 - 
139)


Again, I'm not sure what you're fixing here.  Every execution command is 
going to have the password properties in them anyway, and there will be 
hundreds in /var/lib/ambari-agent/data.


- Nate Cole


On Sept. 12, 2016, 6:48 p.m., Anita Jebaraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51705/
> ---
> 
> (Updated Sept. 12, 2016, 6:48 p.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-18334
> https://issues.apache.org/jira/browse/AMBARI-18334
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The configurations.json file loaded in the ambari-agent cache located at 
> /var/lib/ambari-agent/cache/cluster_configuration contains password details 
> in plaintext (Ex: ssl.client.keystore.password,ssl.client.truststore.password 
> etc.). The values are loaded both in the memory cache and file cache, the 
> file seems to be used only for debugging purposes, so it would be a better 
> approach to mask the passwords in the file.
> 
> Approach:
> 
> The password_config_type is included in the heartbeat response for alert 
> definition command and execution command, for which the values are dumped 
> into the json file. The password_config_type contains the information on 
> which properties in the configurations has the propertyType password. Based 
> on the response, the json is parsed and the password values are masked before 
> dumping it into the configurations.json file.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py 72b87be 
>   ambari-agent/src/test/python/ambari_agent/TestAlerts.py 2bddc43 
>   ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 
> a418f6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java
>  0562c15 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/AlertDefinitionCommand.java
>  4d2e048 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  29737ee 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 70c24f9 
> 
> Diff: https://reviews.apache.org/r/51705/diff/
> 
> 
> Testing
> ---
> 
> Updated the test cases.
> Ran mvn test.
> 
> Manually tested by setting up a cluster, the password fields in the 
> configurations.json is masked. During testing, everytime the ambari agent is 
> restarted, it registers with the server and the memory cache and file cache 
> are updated, the alerts in turn uses the value from the memory cache.
> 
> 
> Thanks,
> 
> Anita Jebaraj
> 
>

Review Request 51844: Audit logs should not be filtered by Log Feeder

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51844/
---

Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.


Bugs: AMBARI-18372
https://issues.apache.org/jira/browse/AMBARI-18372


Repository: ambari


Description
---

LogFeeder shouldn't filter audit logs.


Diffs
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/FilterLogData.java
 801a289 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
 2c81c19 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandlerTest.java
 02ffd47 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputManagerTest.java
 e103346 

Diff: https://reviews.apache.org/r/51844/diff/


Testing
---

Tested on local cluster


Thanks,

Miklos Gergely

Re: Review Request 51818: YAML Maps For Storm Are Not Being Escaped Correctly

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51818/#review148708
---


Ship it!





ambari-common/src/main/python/ambari_commons/yaml_utils.py (lines 22 - 28)


NOW I know who to go to for regex shenanigans ;)


- Nate Cole


On Sept. 12, 2016, 3:13 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51818/
> ---
> 
> (Updated Sept. 12, 2016, 3:13 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Robert Levas.
> 
> 
> Bugs: AMBARI-18366
> https://issues.apache.org/jira/browse/AMBARI-18366
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When attempting to add a custom {{storm-site}} property which is a YAML map 
> of maps, the {{storm.yaml}} file being generated is incorrectly escaping the 
> map. 
> 
> {code:title="Expected YAML"}
> nimbus.impersonation.acl:
> storm:
>   hosts:
> [c6401.ambari.apache.org, c6402.ambari.apache.org]
>   groups:
> [hadoop, foo]
> {code}
> 
> {code:title="Actual YAML"}
> nimbus.impersonation.acl: 'storm:
>   hosts:
> [c6401.ambari.apache.org, c6402.ambari.apache.org]
>   groups:
> [hadoop, foo]'
> {code}
> 
> - We should not be escaping YAML maps
> - YAML maps must being on a newline with no leading whitespace
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/ambari_commons/yaml_utils.py bb05c8a 
>   ambari-server/src/test/python/TestYAMLUtils.py bdbb11f 
> 
> Diff: https://reviews.apache.org/r/51818/diff/
> 
> 
> Testing
> ---
> 
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51238/#review148709
---


Ship it!




Ship It!

- Nate Cole


On Sept. 13, 2016, 5:31 a.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51238/
> ---
> 
> (Updated Sept. 13, 2016, 5:31 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-15538
> https://issues.apache.org/jira/browse/AMBARI-15538
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Implementation contains the following things:
> - On startup, add-on service repos are loaded into the stack model
> - On startup, add-on service repos are merged into the VDF's downloaded by 
> LatestRepoCallable
> - On startup, if the is an existing cluster, it's repository version entity 
> is potentially updated with new add-on service repos.
> - Repository definitions contain two new optional fields: service_name and 
> service_version. (null for stack repositories) 
> - Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5)
> - New unit tests
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
>  7403dba 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  1d82928 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  bae4ab6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
>  02fc2ec 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> bc94104 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java
>  3acc617 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 1fee589 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  7bcd08b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java
>  83a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> 14ff9de 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
>  d26b834 
>   ambari-server/src/main/resources/version_definition.xsd bd49028 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java
>  1d73ff3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/common-services/ADDON/1.0/configuration/addon-env.xml
>  PRE-CREATION 
>   ambari-server/src/test/resources/common-services/ADDON/1.0/metainfo.xml 
> PRE-CREATION 
>   
> ambari-server/src/test/resources/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest_initialRepos.json
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/repos/repoinfo.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/configuration/microsoft-r-env.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/microsoft_r.py
>  61ea96b 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/custom-services/MICROSOFT_R/8.0.0/repos/repoinfo.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51238/diff/
> 
> 
> Testing
> ---
> 
> - Manually tested
> - Wrote new unit tests
> - All unit tests passed except two which were failing in CI builds as well.
> 
> 
> Thanks,
> 
> Balázs Bence Sári
> 
>

Re: Review Request 51844: Audit logs should not be filtered by Log Feeder

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51844/#review148717
---


Ship it!




Ship It!

- Robert Nettleton


On Sept. 13, 2016, 12:37 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51844/
> ---
> 
> (Updated Sept. 13, 2016, 12:37 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18372
> https://issues.apache.org/jira/browse/AMBARI-18372
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> LogFeeder shouldn't filter audit logs.
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/FilterLogData.java
>  801a289 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
>  2c81c19 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandlerTest.java
>  02ffd47 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputManagerTest.java
>  e103346 
> 
> Diff: https://reviews.apache.org/r/51844/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 51725: Microsoft-R client should work in a secured cluster

[Balázs Bence Sári]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51725/
---

(Updated Sept. 13, 2016, 1:29 p.m.)


Review request for Ambari, Attila Doroszlai, Jayush Luniya, Oliver Szabo, Sumit 
Mohanty, and Sebastian Toader.


Changes
---

Fixed Attila's findings. Removed unused host_sys_prepped declaration from 
params.py


Bugs: AMBARI-18338
https://issues.apache.org/jira/browse/AMBARI-18338


Repository: ambari


Description
---

- Added kerberos json to make sure the client can use the smokeuser and hdfs 
keytabs. 
- Added the creation of /user/RevoShare hdfs directory to the service check 
(with hdfs user/principal)
- Extended stack support to HDP 2.4 and 2.5


Diffs (updated)
-

  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/kerberos.json
 PRE-CREATION 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params.py
 36541e7 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params_linux.py
 d72c42e 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/service_check.py
 8a425ca 

Diff: https://reviews.apache.org/r/51725/diff/


Testing
---

Manually tested the installation and kerberization of MSFT-R via UI and 
blueprints on 1-8 node clusters.


Thanks,

Balázs Bence Sári

Review Request 51846: AMBARI-18375 Ranger Plugin configs is not generated for Hive interactive

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51846/
---

Review request for Ambari, Gautam Borad, Jayush Luniya, Sumit Mohanty, and 
Velmurugan Periasamy.


Bugs: AMBARI-18375
https://issues.apache.org/jira/browse/AMBARI-18375


Repository: ambari


Description
---

If audit to hdfs is OFF for ranger hive plugin, configs related to plugins are 
not generated for Hive interactive


Diffs
-

  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py
 0b5d5db 

Diff: https://reviews.apache.org/r/51846/diff/


Testing
---

Tested Ranger hive plugin and hive interactive with Ranger install on centos6


Thanks,

Mugdha Varadkar

Re: Review Request 51818: YAML Maps For Storm Are Not Being Escaped Correctly

[Jonathan Hurley]

> On Sept. 13, 2016, 8:48 a.m., Nate Cole wrote:
> > ambari-common/src/main/python/ambari_commons/yaml_utils.py, lines 22-28
> > 
> >
> > NOW I know who to go to for regex shenanigans ;)

I always say, "If you need to use regex, then you're doing it wrong" :)


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51818/#review148708
---


On Sept. 12, 2016, 3:13 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51818/
> ---
> 
> (Updated Sept. 12, 2016, 3:13 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Robert Levas.
> 
> 
> Bugs: AMBARI-18366
> https://issues.apache.org/jira/browse/AMBARI-18366
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When attempting to add a custom {{storm-site}} property which is a YAML map 
> of maps, the {{storm.yaml}} file being generated is incorrectly escaping the 
> map. 
> 
> {code:title="Expected YAML"}
> nimbus.impersonation.acl:
> storm:
>   hosts:
> [c6401.ambari.apache.org, c6402.ambari.apache.org]
>   groups:
> [hadoop, foo]
> {code}
> 
> {code:title="Actual YAML"}
> nimbus.impersonation.acl: 'storm:
>   hosts:
> [c6401.ambari.apache.org, c6402.ambari.apache.org]
>   groups:
> [hadoop, foo]'
> {code}
> 
> - We should not be escaping YAML maps
> - YAML maps must being on a newline with no leading whitespace
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/ambari_commons/yaml_utils.py bb05c8a 
>   ambari-server/src/test/python/TestYAMLUtils.py bdbb11f 
> 
> Diff: https://reviews.apache.org/r/51818/diff/
> 
> 
> Testing
> ---
> 
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 51846: AMBARI-18375 Ranger Plugin configs is not generated for Hive interactive

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51846/#review148720
---


Ship it!




Ship It!

- Sumit Mohanty


On Sept. 13, 2016, 1:39 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51846/
> ---
> 
> (Updated Sept. 13, 2016, 1:39 p.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jayush Luniya, Sumit Mohanty, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18375
> https://issues.apache.org/jira/browse/AMBARI-18375
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If audit to hdfs is OFF for ranger hive plugin, configs related to plugins 
> are not generated for Hive interactive
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py
>  0b5d5db 
> 
> Diff: https://reviews.apache.org/r/51846/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger hive plugin and hive interactive with Ranger install on centos6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 51815: Introduce conditional dependencies in stack defition to handle blueprint validation gracefully

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51815/#review148721
---




ambari-server/src/main/java/org/apache/ambari/server/state/DependencyConditionInfo.java
 (line 103)


do you support other type of checks ? such as the property has to be a 
certain value ?



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
(line 55)


this is already a subsection of "dependency", it can just be called 
"conditions"



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
(line 56)


same here, no need to reiterate it's for "dependency"



ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
(line 58)


what does "dfs.nameservices" mean in this conditional 
check? as a negative test, what if a user manually added the property to 
hdfs-site.xml ?


- Di Li


On Sept. 12, 2016, 5:45 p.m., Amruta Borkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51815/
> ---
> 
> (Updated Sept. 12, 2016, 5:45 p.m.)
> 
> 
> Review request for Ambari, Di Li, Juanjo  Marron, Laszlo Puskas, and Robert 
> Nettleton.
> 
> 
> Bugs: AMBARI-18355
> https://issues.apache.org/jira/browse/AMBARI-18355
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently stack definitions do not list conditional dependencies, adding 
> those to the stack definitions would make it easy to validate errors in case 
> of blueprint deployment. Please refer to document attached to Jira
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/DependencyConditionInfo.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/DependencyInfo.java
>  e3db662 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
>  a5f33ff 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> 65d166a 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java
>  b1de8ef 
> 
> Diff: https://reviews.apache.org/r/51815/diff/
> 
> 
> Testing
> ---
> 
> Written Junit test cases. Perfomred manual testing to check the Namenode HA 
> component dependency. Was able to proceed with the installation for valid 
> blueprint. and got validation error message while registering blueprint when 
> the Blueprint did not satisfy the conditional dependencies.
> 
> 
> Thanks,
> 
> Amruta Borkar
> 
>

Re: Review Request 51815: Introduce conditional dependencies in stack defition to handle blueprint validation gracefully

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51815/#review148722
---



Please add Shantanu as a reviewer

- Di Li


On Sept. 12, 2016, 5:45 p.m., Amruta Borkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51815/
> ---
> 
> (Updated Sept. 12, 2016, 5:45 p.m.)
> 
> 
> Review request for Ambari, Di Li, Juanjo  Marron, Laszlo Puskas, and Robert 
> Nettleton.
> 
> 
> Bugs: AMBARI-18355
> https://issues.apache.org/jira/browse/AMBARI-18355
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently stack definitions do not list conditional dependencies, adding 
> those to the stack definitions would make it easy to validate errors in case 
> of blueprint deployment. Please refer to document attached to Jira
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/DependencyConditionInfo.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/DependencyInfo.java
>  e3db662 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
>  a5f33ff 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> 65d166a 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java
>  b1de8ef 
> 
> Diff: https://reviews.apache.org/r/51815/diff/
> 
> 
> Testing
> ---
> 
> Written Junit test cases. Perfomred manual testing to check the Namenode HA 
> component dependency. Was able to proceed with the installation for valid 
> blueprint. and got validation error message while registering blueprint when 
> the Blueprint did not satisfy the conditional dependencies.
> 
> 
> Thanks,
> 
> Amruta Borkar
> 
>

Re: Review Request 49804: improve resourcemanager HA description

[Aleksandr Kovalenko]

> On Сен. 12, 2016, 9:08 д.п., Aleksandr Kovalenko wrote:
> > ambari-web/app/models/host_component.js, line 333
> > 
> >
> > What is the reason for this change?
> > Why we decide to not show Enable label if HA is already enabled?
> 
> wang yaoxin wrote:
> Thank you ,Aleksandr . Just like the NameNode HA , it's better to decide 
> the RM Enable lable and cssClass

I see, thanks.


- Aleksandr


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49804/#review148468
---


On Сен. 12, 2016, 3:24 д.п., wang yaoxin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49804/
> ---
> 
> (Updated Сен. 12, 2016, 3:24 д.п.)
> 
> 
> Review request for Ambari, Aleksandr Kovalenko, jun aoki, Zhe (Joe) Wang, 
> Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17518
> https://issues.apache.org/jira/browse/AMBARI-17518
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> improve the resourcemanager HA description
> 
> 
> Diffs
> -
> 
>   ambari-web/app/models/host_component.js 973b809 
> 
> Diff: https://reviews.apache.org/r/49804/diff/
> 
> 
> Testing
> ---
> 
> done
> 
> 
> Thanks,
> 
> wang yaoxin
> 
>

Re: Review Request 49804: improve resourcemanager HA description

[Aleksandr Kovalenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49804/#review148726
---


Ship it!




Ship It!

- Aleksandr Kovalenko


On Сен. 12, 2016, 3:24 д.п., wang yaoxin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49804/
> ---
> 
> (Updated Сен. 12, 2016, 3:24 д.п.)
> 
> 
> Review request for Ambari, Aleksandr Kovalenko, jun aoki, Zhe (Joe) Wang, 
> Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17518
> https://issues.apache.org/jira/browse/AMBARI-17518
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> improve the resourcemanager HA description
> 
> 
> Diffs
> -
> 
>   ambari-web/app/models/host_component.js 973b809 
> 
> Diff: https://reviews.apache.org/r/49804/diff/
> 
> 
> Testing
> ---
> 
> done
> 
> 
> Thanks,
> 
> wang yaoxin
> 
>

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Robert Levas]

> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 5292-5294
> > 
> >
> > Why not just return here which would prevent you from needing to embed 
> > the rest of the logic in the if-statement

Then I would have 2 return statements... but I can make the chance since most 
developers like to short-curcuit their code.


> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 5298-5301
> > 
> >
> > KERBEROS_AUTH_USER_TYPES has a default value, so `getProperty` will 
> > never return null.

This is a safety measure in the event someone changes the definition of 
`KERBEROS_AUTH_USER_TYPES`. Plus I generally feel the need to check for null 
and empty in these cases. But I can remove this check.


> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 5310-5311
> > 
> >
> > Invalid types would seem to be a mis-configuration which could cause 
> > connectivity issues with Ambari; did you want to instead throw a real 
> > exception here?

ok.. that is an option.


> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 5345-5348
> > 
> >
> > Same question as above - because these configurations involve 
> > connection to Ambari, should we instead throw exceptions and fail startup 
> > on problems like this?

ok.. will fix.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148699
---


On Sept. 13, 2016, 5 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 5 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:51:23.081s
> [INFO] Finished at: T

Re: Review Request 51725: Microsoft-R client should work in a secured cluster

[Balázs Bence Sári]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51725/
---

(Updated Sept. 13, 2016, 3:55 p.m.)


Review request for Ambari, Attila Doroszlai, Jayush Luniya, Oliver Szabo, 
Robert Levas, Sandor Magyari, Sumit Mohanty, and Sebastian Toader.


Changes
---

Re-added host_sys_prepped


Bugs: AMBARI-18338
https://issues.apache.org/jira/browse/AMBARI-18338


Repository: ambari


Description
---

- Added kerberos json to make sure the client can use the smokeuser and hdfs 
keytabs. 
- Added the creation of /user/RevoShare hdfs directory to the service check 
(with hdfs user/principal)
- Extended stack support to HDP 2.4 and 2.5


Diffs (updated)
-

  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/kerberos.json
 PRE-CREATION 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params.py
 36541e7 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params_linux.py
 d72c42e 
  
contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/service_check.py
 8a425ca 

Diff: https://reviews.apache.org/r/51725/diff/


Testing
---

Manually tested the installation and kerberization of MSFT-R via UI and 
blueprints on 1-8 node clusters.


Thanks,

Balázs Bence Sári

Re: Review Request 51838: Do not execute stack-selector-tool after HDP component install on sysprepped hosts.

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51838/#review148735
---


Ship it!




I believe that the original case for this `hdp-select set all` was when adding 
a host back to the cluster after an upgrade. Because prior symlinks existed, 
new components will not have their symlink pointers created by RPM. We should 
make it explicit, perhaps, in a logged message that this is the side-effect of 
skipping this hdp-select set all call. 

However, we should be allowed to invoke `hdp-select set all` as many times as 
we want, especially since it's being done within a lock. Perhaps there needs to 
be a Jira to track why it's failing and have it gracefully recover.

- Jonathan Hurley


On Sept. 13, 2016, 9:22 a.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51838/
> ---
> 
> (Updated Sept. 13, 2016, 9:22 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Jayush Luniya, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18370
> https://issues.apache.org/jira/browse/AMBARI-18370
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1. Modified Ambaripreupload.py to invoke 'hdp-select set all' to ensure that 
> when preparing a sysprepped host the stack selector tool was executed for the 
> selected HDP components
> 2. The shared initialisation script that is executed once after each HDP 
> component install as AFTER-install hook skips executing 'hdp-select set all' 
> if it's a sysprepped host
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  819d8f7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  9982dc6 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
>  06a366e 
> 
> Diff: https://reviews.apache.org/r/51838/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Python unit tests:
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:43.849s
> [INFO] Finished at: Tue Sep 13 09:56:49 CEST 2016
> [INFO] Final Memory: 73M/799M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 51705: Password in the configurations.json file in the ambari-agent cache is not encrypted

[Anita Jebaraj]

> On Sept. 13, 2016, 12:37 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py, lines 
> > 138-140
> > 
> >
> > Again, I'm not sure what you're fixing here.  Every execution command 
> > is going to have the password properties in them anyway, and there will be 
> > hundreds in /var/lib/ambari-agent/data.

Hi Nate, the idea here is to avoid using plaintext passwords in the files that 
are saved in the ambari agent cache. To begin with I am trying to come up with 
an approach to mask the passwords in the configurations.json, since it is used 
only for debugging purposes and hence no impact is expected. If the approach is 
accepted, I would like to continue the work of masking the passwords in all the 
files in the agent side by creating new JIRAs.


- Anita


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51705/#review148705
---


On Sept. 12, 2016, 10:48 p.m., Anita Jebaraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51705/
> ---
> 
> (Updated Sept. 12, 2016, 10:48 p.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-18334
> https://issues.apache.org/jira/browse/AMBARI-18334
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The configurations.json file loaded in the ambari-agent cache located at 
> /var/lib/ambari-agent/cache/cluster_configuration contains password details 
> in plaintext (Ex: ssl.client.keystore.password,ssl.client.truststore.password 
> etc.). The values are loaded both in the memory cache and file cache, the 
> file seems to be used only for debugging purposes, so it would be a better 
> approach to mask the passwords in the file.
> 
> Approach:
> 
> The password_config_type is included in the heartbeat response for alert 
> definition command and execution command, for which the values are dumped 
> into the json file. The password_config_type contains the information on 
> which properties in the configurations has the propertyType password. Based 
> on the response, the json is parsed and the password values are masked before 
> dumping it into the configurations.json file.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py 72b87be 
>   ambari-agent/src/test/python/ambari_agent/TestAlerts.py 2bddc43 
>   ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 
> a418f6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java
>  0562c15 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/AlertDefinitionCommand.java
>  4d2e048 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  29737ee 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 70c24f9 
> 
> Diff: https://reviews.apache.org/r/51705/diff/
> 
> 
> Testing
> ---
> 
> Updated the test cases.
> Ran mvn test.
> 
> Manually tested by setting up a cluster, the password fields in the 
> configurations.json is masked. During testing, everytime the ambari agent is 
> restarted, it registers with the server and the memory cache and file cache 
> are updated, the alerts in turn uses the value from the memory cache.
> 
> 
> Thanks,
> 
> Anita Jebaraj
> 
>

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Robert Levas]

> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  line 2586
> > 
> >
> > Although I understand the motivation, I still don't like doing this 
> > since getting the backing Properties object for the entire Configuration 
> > would be missing these properties. Can we mirror them in the normal 
> > Properties object as well?

I am not sure what you mean.  The backing properties object contains the 
properties from the ambari.properties file and is being used to fill the 
KerberosAuthenticationProperties instance.   Does this data need to be copied 
to another map?


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148699
---


On Sept. 13, 2016, 5 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 5 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:51:23.081s
> [INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
> [INFO] Final Memory: 48M/1792M
> [INFO] 
> 
> 
> # Jenkins test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 01:34 h
> [INFO] Finished at: 2016-09-13T08:46:23+00:00
> [INFO] Final Memory: 178M/2672M
> [INFO] 
> 
> 
> {color:green}+1 overall{color}.  Here are the results of testing the latest 
> attachment 
>   
> http://issues.apache.org/jira/secure/attachment/12828101/AMBARI-18365_trunk_01.patch
>   against trunk revision .
> 
> {color:green}+1 @author{color}.  The patch does not contain any @author 
> tags.
> 
> {color:green}+1 tests included{color}.  The patch appears to include 2 
> new or modified test files.
> 
> {color:green}+1 javac{color}.  The applied patch does not increase the 
> total number of javac compiler warnings.
> 
> {color:green}+1 release audit{color}.  The applied patch does not 
> increase the total number of 

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Robert Levas]

> On Sept. 13, 2016, 8:29 a.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 1334-1336
> > 
> >
> > Is this specifically for SPNEGO, or any Kerberos auth?  If so, then the 
> > property should use the word "spnego" somewhere like you have for other 
> > properties.

These properties are specificly for Kerberos authentication via SPNEGO - 
*S*imple and *P*rotected GSSAPI *Nego*tiation Mechanism.  SPNEGO can be used 
for other authentication protocols, but is most commponly used for Kerberos 
authencation.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148701
---


On Sept. 13, 2016, 5 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 5 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:51:23.081s
> [INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
> [INFO] Final Memory: 48M/1792M
> [INFO] 
> 
> 
> # Jenkins test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 01:34 h
> [INFO] Finished at: 2016-09-13T08:46:23+00:00
> [INFO] Final Memory: 178M/2672M
> [INFO] 
> 
> 
> {color:green}+1 overall{color}.  Here are the results of testing the latest 
> attachment 
>   
> http://issues.apache.org/jira/secure/attachment/12828101/AMBARI-18365_trunk_01.patch
>   against trunk revision .
> 
> {color:green}+1 @author{color}.  The patch does not contain any @author 
> tags.
> 
> {color:green}+1 tests included{color}.  The patch appears to include 2 
> new or modified test files.
> 
> {color:green}+1 javac{color}.  The applied patch does not increase the 
> total number of javac compiler warnings.
> 
> {color:green}+1 release audit{color}.  The applied patch does not 
> increase the total number of release audit warnings.
> 
> {color:green}+1 core tests{color}.  The patch passed

Re: Review Request 51705: Password in the configurations.json file in the ambari-agent cache is not encrypted

[Nate Cole]

> On Sept. 13, 2016, 8:37 a.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py, lines 
> > 138-140
> > 
> >
> > Again, I'm not sure what you're fixing here.  Every execution command 
> > is going to have the password properties in them anyway, and there will be 
> > hundreds in /var/lib/ambari-agent/data.
> 
> Anita Jebaraj wrote:
> Hi Nate, the idea here is to avoid using plaintext passwords in the files 
> that are saved in the ambari agent cache. To begin with I am trying to come 
> up with an approach to mask the passwords in the configurations.json, since 
> it is used only for debugging purposes and hence no impact is expected. If 
> the approach is accepted, I would like to continue the work of masking the 
> passwords in all the files in the agent side by creating new JIRAs.

That's what I'm saying - since the stack scripts run in their OWN process 
separate from the agent, you can't mask them on serialization.  Flow is:
- Server sends down commands, with passwords in configuration.
- Agent serializes command to disk with passwords
- Agent invokes stack script
- Stack script reads command file with passwords in it

That second step is the important one; we can't mask and persist passwords that 
haven't yet been read.  You're right in that alerts may use only the in-memory 
representation, but that strategy won't work for other stack scripts.


- Nate


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51705/#review148705
---


On Sept. 12, 2016, 6:48 p.m., Anita Jebaraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51705/
> ---
> 
> (Updated Sept. 12, 2016, 6:48 p.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-18334
> https://issues.apache.org/jira/browse/AMBARI-18334
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The configurations.json file loaded in the ambari-agent cache located at 
> /var/lib/ambari-agent/cache/cluster_configuration contains password details 
> in plaintext (Ex: ssl.client.keystore.password,ssl.client.truststore.password 
> etc.). The values are loaded both in the memory cache and file cache, the 
> file seems to be used only for debugging purposes, so it would be a better 
> approach to mask the passwords in the file.
> 
> Approach:
> 
> The password_config_type is included in the heartbeat response for alert 
> definition command and execution command, for which the values are dumped 
> into the json file. The password_config_type contains the information on 
> which properties in the configurations has the propertyType password. Based 
> on the response, the json is parsed and the password values are masked before 
> dumping it into the configurations.json file.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/ClusterConfiguration.py 72b87be 
>   ambari-agent/src/test/python/ambari_agent/TestAlerts.py 2bddc43 
>   ambari-agent/src/test/python/ambari_agent/TestClusterConfigurationCache.py 
> a418f6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java
>  0562c15 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/AlertDefinitionCommand.java
>  4d2e048 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  29737ee 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 70c24f9 
> 
> Diff: https://reviews.apache.org/r/51705/diff/
> 
> 
> Testing
> ---
> 
> Updated the test cases.
> Ran mvn test.
> 
> Manually tested by setting up a cluster, the password fields in the 
> configurations.json is masked. During testing, everytime the ambari agent is 
> restarted, it registers with the server and the memory cache and file cache 
> are updated, the alerts in turn uses the value from the memory cache.
> 
> 
> Thanks,
> 
> Anita Jebaraj
> 
>

Re: Review Request 51725: Microsoft-R client should work in a secured cluster

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51725/#review148751
---


Ship it!




I am concerned with the use of the HDFS *admin* user.  Why not use the revo(?) 
user's identity instead?


contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params_linux.py
 (lines 65 - 79)


Does the HDFS *admin* user really need to perform these operations?


- Robert Levas


On Sept. 13, 2016, 11:55 a.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51725/
> ---
> 
> (Updated Sept. 13, 2016, 11:55 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jayush Luniya, Oliver Szabo, 
> Robert Levas, Sandor Magyari, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-18338
> https://issues.apache.org/jira/browse/AMBARI-18338
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - Added kerberos json to make sure the client can use the smokeuser and hdfs 
> keytabs. 
> - Added the creation of /user/RevoShare hdfs directory to the service check 
> (with hdfs user/principal)
> - Extended stack support to HDP 2.4 and 2.5
> 
> 
> Diffs
> -
> 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/kerberos.json
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params.py
>  36541e7 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params_linux.py
>  d72c42e 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/service_check.py
>  8a425ca 
> 
> Diff: https://reviews.apache.org/r/51725/diff/
> 
> 
> Testing
> ---
> 
> Manually tested the installation and kerberization of MSFT-R via UI and 
> blueprints on 1-8 node clusters.
> 
> 
> Thanks,
> 
> Balázs Bence Sári
> 
>

Re: Review Request 51838: Do not execute stack-selector-tool after HDP component install on sysprepped hosts.

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51838/#review148753
---


Ship it!




Ship It!

- Jayush Luniya


On Sept. 13, 2016, 1:22 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51838/
> ---
> 
> (Updated Sept. 13, 2016, 1:22 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Jayush Luniya, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18370
> https://issues.apache.org/jira/browse/AMBARI-18370
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1. Modified Ambaripreupload.py to invoke 'hdp-select set all' to ensure that 
> when preparing a sysprepped host the stack selector tool was executed for the 
> selected HDP components
> 2. The shared initialisation script that is executed once after each HDP 
> component install as AFTER-install hook skips executing 'hdp-select set all' 
> if it's a sysprepped host
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  819d8f7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  9982dc6 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
>  06a366e 
> 
> Diff: https://reviews.apache.org/r/51838/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Python unit tests:
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:43.849s
> [INFO] Finished at: Tue Sep 13 09:56:49 CEST 2016
> [INFO] Final Memory: 73M/799M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 51725: Microsoft-R client should work in a secured cluster

[Balázs Bence Sári]

> On Sept. 13, 2016, 6:13 p.m., Robert Levas wrote:
> > I am concerned with the use of the HDFS *admin* user.  Why not use the 
> > revo(?) user's identity instead?

Revo doesn't have its own user. We need to ensure that the /user/RevoShare 
exists on hdfs, that's what we need the HDFS admin user for.
The /user/RevoShare is the base folder for individual revo users, their revo 
home folders will be placed here. The folder has to have r/w rights for 
everyone.


- Balázs Bence


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51725/#review148751
---


On Sept. 13, 2016, 3:55 p.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51725/
> ---
> 
> (Updated Sept. 13, 2016, 3:55 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jayush Luniya, Oliver Szabo, 
> Robert Levas, Sandor Magyari, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-18338
> https://issues.apache.org/jira/browse/AMBARI-18338
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - Added kerberos json to make sure the client can use the smokeuser and hdfs 
> keytabs. 
> - Added the creation of /user/RevoShare hdfs directory to the service check 
> (with hdfs user/principal)
> - Extended stack support to HDP 2.4 and 2.5
> 
> 
> Diffs
> -
> 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/kerberos.json
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params.py
>  36541e7 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/params_linux.py
>  d72c42e 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/package/scripts/service_check.py
>  8a425ca 
> 
> Diff: https://reviews.apache.org/r/51725/diff/
> 
> 
> Testing
> ---
> 
> Manually tested the installation and kerberization of MSFT-R via UI and 
> blueprints on 1-8 node clusters.
> 
> 
> Thanks,
> 
> Balázs Bence Sári
> 
>

Review Request 51851: AMBARI-18332: Blueprints: API should make available "setting" property from blueprint

[Nahappan Somasundaram]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51851/
---

Review request for Ambari, Robert Nettleton and Sumit Mohanty.


Bugs: AMBARI-18332
https://issues.apache.org/jira/browse/AMBARI-18332


Repository: ambari


Description
---

AMBARI-18332: Blueprints: API should make available "setting" property from 
blueprint

** Issue: **
A new section *setting* was added to blueprints to support auto-start. However, 
when querying the blueprint that used to deploy a cluster, the *setting* 
section is absent.

** Fix: **
Add the *setting* section to the blueprint response that is returned during a 
http://:/api/v1/blueprints/


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintResourceProvider.java
 775827b4425a6668fd11d435ca7e4175d1f75d16 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintResourceProviderTest.java
 75eff7e1d1d5b8db232f109eb08b6cb57ce0a105 

Diff: https://reviews.apache.org/r/51851/diff/


Testing
---

** 1. mvn clean install **
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ... SUCCESS [11.486s]
[INFO] Apache Ambari Project POM . SUCCESS [0.041s]
[INFO] Ambari Web  SUCCESS [1:11.461s]
[INFO] Ambari Views .. SUCCESS [1.161s]
[INFO] Ambari Admin View . SUCCESS [8.439s]
[INFO] ambari-metrics  SUCCESS [0.681s]
[INFO] Ambari Metrics Common . SUCCESS [4.584s]
[INFO] Ambari Metrics Hadoop Sink  SUCCESS [2.078s]
[INFO] Ambari Metrics Flume Sink . SUCCESS [1.289s]
[INFO] Ambari Metrics Kafka Sink . SUCCESS [1.410s]
[INFO] Ambari Metrics Storm Sink . SUCCESS [4.003s]
[INFO] Ambari Metrics Storm Sink (Legacy)  SUCCESS [1.559s]
[INFO] Ambari Metrics Collector .. SUCCESS [10.463s]
[INFO] Ambari Metrics Monitor  SUCCESS [1.870s]
[INFO] Ambari Metrics Grafana  SUCCESS [1.009s]
[INFO] Ambari Metrics Assembly ... SUCCESS [1:19.907s]
[INFO] Ambari Server . SUCCESS [3:03.887s]
[INFO] Ambari Functional Tests ... SUCCESS [2.286s]
[INFO] Ambari Agent .. SUCCESS [26.876s]
[INFO] Ambari Client . SUCCESS [0.048s]
[INFO] Ambari Python Client .. SUCCESS [0.940s]
[INFO] Ambari Groovy Client .. SUCCESS [2.030s]
[INFO] Ambari Shell .. SUCCESS [0.047s]
[INFO] Ambari Python Shell ... SUCCESS [0.672s]
[INFO] Ambari Groovy Shell ... SUCCESS [0.994s]
[INFO] ambari-logsearch .. SUCCESS [0.302s]
[INFO] Ambari Logsearch Appender . SUCCESS [0.225s]
[INFO] Ambari Logsearch Solr Client .. SUCCESS [1.219s]
[INFO] Ambari Logsearch Portal ... SUCCESS [6.735s]
[INFO] Ambari Logsearch Log Feeder ... SUCCESS [3.926s]
[INFO] Ambari Logsearch Assembly . SUCCESS [0.074s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 7:13.702s
[INFO] Finished at: Tue Sep 13 11:00:07 PDT 2016
[INFO] Final Memory: 302M/1191M
[INFO] 

** 2. mvn test -DskipPythonTests -Dtest=Blueprint*Test **

** Added a new test BlueprintResourceProviderTest::testPopulateSettingList() **
* This test creates the setting JSON and adds it to the blueprint response *

Results :

Tests run: 247, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 28.790s
[INFO] Finished at: Tue Sep 13 11:13:16 PDT 2016
[INFO] Final Memory: 51M/786M
[INFO] 

** 3. Manual Tests **

* Set up a VM with the latest trunk build patched with ambari-server JAR from 
the local build.
* Applied a blueprint: curl -u admin:admin -H "X-Requested-By: ambari" -X POST 
-d @./bp1.json http://c6401.ambari.apache.org:8080/api/v1/blueprints/bp1
* Ran curl -u admin:

Re: Review Request 51838: Do not execute stack-selector-tool after HDP component install on sysprepped hosts.

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51838/#review148757
---




ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
 (line 49)


Has this been tested on a sysprepped cluster that is deployed and then 
undergoes either one of these 2 operations:
* add another service
* perform an EU/RU (this may not be supported yet, but we need to be aware 
that we're using the host_sysprepped_flag to indicate a fresh install but the 
code still runs after an EU/RU).

If those ops are not supported in sysprepped clusters, then close this 
issue.


- Alejandro Fernandez


On Sept. 13, 2016, 1:22 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51838/
> ---
> 
> (Updated Sept. 13, 2016, 1:22 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Jayush Luniya, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18370
> https://issues.apache.org/jira/browse/AMBARI-18370
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1. Modified Ambaripreupload.py to invoke 'hdp-select set all' to ensure that 
> when preparing a sysprepped host the stack selector tool was executed for the 
> selected HDP components
> 2. The shared initialisation script that is executed once after each HDP 
> component install as AFTER-install hook skips executing 'hdp-select set all' 
> if it's a sysprepped host
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  819d8f7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  9982dc6 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
>  06a366e 
> 
> Diff: https://reviews.apache.org/r/51838/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Python unit tests:
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:43.849s
> [INFO] Finished at: Tue Sep 13 09:56:49 CEST 2016
> [INFO] Final Memory: 73M/799M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 51838: Do not execute stack-selector-tool after HDP component install on sysprepped hosts.

[Alejandro Fernandez]

> On Sept. 13, 2016, 4:34 p.m., Jonathan Hurley wrote:
> > I believe that the original case for this `hdp-select set all` was when 
> > adding a host back to the cluster after an upgrade. Because prior symlinks 
> > existed, new components will not have their symlink pointers created by 
> > RPM. We should make it explicit, perhaps, in a logged message that this is 
> > the side-effect of skipping this hdp-select set all call. 
> > 
> > However, we should be allowed to invoke `hdp-select set all` as many times 
> > as we want, especially since it's being done within a lock. Perhaps there 
> > needs to be a Jira to track why it's failing and have it gracefully recover.

I agree with Jonathan, if the symlinks are correct, "hdp-select set all" should 
be a no-op. I don't want to use this as a scapegoat for another underlying 
issue that we haven't discovered yet.


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51838/#review148735
---


On Sept. 13, 2016, 1:22 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51838/
> ---
> 
> (Updated Sept. 13, 2016, 1:22 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Jayush Luniya, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18370
> https://issues.apache.org/jira/browse/AMBARI-18370
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1. Modified Ambaripreupload.py to invoke 'hdp-select set all' to ensure that 
> when preparing a sysprepped host the stack selector tool was executed for the 
> selected HDP components
> 2. The shared initialisation script that is executed once after each HDP 
> component install as AFTER-install hook skips executing 'hdp-select set all' 
> if it's a sysprepped host
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  819d8f7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  9982dc6 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
>  06a366e 
> 
> Diff: https://reviews.apache.org/r/51838/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Python unit tests:
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:43.849s
> [INFO] Finished at: Tue Sep 13 09:56:49 CEST 2016
> [INFO] Final Memory: 73M/799M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 51838: Do not execute stack-selector-tool after HDP component install on sysprepped hosts.

[Jayush Luniya]

> On Sept. 13, 2016, 6:42 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py,
> >  line 49
> > 
> >
> > Has this been tested on a sysprepped cluster that is deployed and then 
> > undergoes either one of these 2 operations:
> > * add another service
> > * perform an EU/RU (this may not be supported yet, but we need to be 
> > aware that we're using the host_sysprepped_flag to indicate a fresh install 
> > but the code still runs after an EU/RU).
> > 
> > If those ops are not supported in sysprepped clusters, then close this 
> > issue.

Wouldnt we be calling install_packages() during RU/EU? The after install hook 
wouldnt be called during RU/EU.


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51838/#review148757
---


On Sept. 13, 2016, 1:22 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51838/
> ---
> 
> (Updated Sept. 13, 2016, 1:22 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Jayush Luniya, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18370
> https://issues.apache.org/jira/browse/AMBARI-18370
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 1. Modified Ambaripreupload.py to invoke 'hdp-select set all' to ensure that 
> when preparing a sysprepped host the stack selector tool was executed for the 
> selected HDP components
> 2. The shared initialisation script that is executed once after each HDP 
> component install as AFTER-install hook skips executing 'hdp-select set all' 
> if it's a sysprepped host
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  819d8f7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  9982dc6 
>   
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
>  06a366e 
> 
> Diff: https://reviews.apache.org/r/51838/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Python unit tests:
> --
> Total run:1125
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:43.849s
> [INFO] Finished at: Tue Sep 13 09:56:49 CEST 2016
> [INFO] Final Memory: 73M/799M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 51846: AMBARI-18375 Ranger Plugin configs is not generated for Hive interactive

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51846/#review148768
---


Ship it!




Ship It!

- Jayush Luniya


On Sept. 13, 2016, 1:39 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51846/
> ---
> 
> (Updated Sept. 13, 2016, 1:39 p.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jayush Luniya, Sumit Mohanty, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18375
> https://issues.apache.org/jira/browse/AMBARI-18375
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If audit to hdfs is OFF for ranger hive plugin, configs related to plugins 
> are not generated for Hive interactive
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py
>  0b5d5db 
> 
> Diff: https://reviews.apache.org/r/51846/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger hive plugin and hive interactive with Ranger install on centos6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 51846: AMBARI-18375 Ranger Plugin configs is not generated for Hive interactive

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51846/#review148772
---




ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py
 


What was the diff?


- Alejandro Fernandez


On Sept. 13, 2016, 1:39 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51846/
> ---
> 
> (Updated Sept. 13, 2016, 1:39 p.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jayush Luniya, Sumit Mohanty, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18375
> https://issues.apache.org/jira/browse/AMBARI-18375
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If audit to hdfs is OFF for ranger hive plugin, configs related to plugins 
> are not generated for Hive interactive
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive_interactive.py
>  0b5d5db 
> 
> Diff: https://reviews.apache.org/r/51846/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger hive plugin and hive interactive with Ranger install on centos6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 51238: Implementation for AMBARI-15538: Support service-specific repo for add-on services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51238/#review148773
---


Ship it!




Verfied following scenarios
1. Add additional repo entry in the stack's repoinfo.xml. This addresses 
AMBARI-17285.
2. Verified adding a repoinfo.xml at custom-service level (service level 
extension point) adds the repo to list of repos for all stacks in the 
hierarchy. (i.e. SOLR service when added to HDP-2.4 as add-on service will also 
lead to the repo to be added to HDP-2.5)
3. Verified that if we have MYSERVICE/1.0 -> HDP-2.4 and MYSERVICE/2.0 -> 
HDP-2.5 and the 2 service versions of the service have different repoinfo.xml 
then the correct repoinfo is added to the respective stacks
4. Verified that repoinfo.xml added to a service definition in an extension is 
processed when the extension is linked and refreshed.


NOTE: When backporting this patch to 2.4/2.5 branches, we will have conflicts 
as the patch for AMBARI-18325 is in trunk, but not in those branches.

- Jayush Luniya


On Sept. 13, 2016, 9:31 a.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51238/
> ---
> 
> (Updated Sept. 13, 2016, 9:31 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Nate Cole, Sumit Mohanty, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-15538
> https://issues.apache.org/jira/browse/AMBARI-15538
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Implementation contains the following things:
> - On startup, add-on service repos are loaded into the stack model
> - On startup, add-on service repos are merged into the VDF's downloaded by 
> LatestRepoCallable
> - On startup, if the is an existing cluster, it's repository version entity 
> is potentially updated with new add-on service repos.
> - Repository definitions contain two new optional fields: service_name and 
> service_version. (null for stack repositories) 
> - Small changes on the Mictrosoft-R mpack (supoorts HDP-2.4 and 2.5)
> - New unit tests
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
>  7403dba 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  1d82928 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  bae4ab6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
>  02fc2ec 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> bc94104 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java
>  3acc617 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 1fee589 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  7bcd08b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartup.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java
>  83a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> 14ff9de 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
>  d26b834 
>   ambari-server/src/main/resources/version_definition.xsd bd49028 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/RepoUtilTest.java 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerCommonServicesTest.java
>  1d73ff3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackModuleTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/common-services/ADDON/1.0/configuration/addon-env.xml
>  PRE-CREATION 
>   ambari-server/src/test/resources/common-services/ADDON/1.0/metainfo.xml 
> PRE-CREATION 
>   
> ambari-server/src/test/resources/org/apache/ambari/server/stack/UpdateActiveRepoVersionOnStartupTest_initialRepos.json
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/ADDON/repos/repoinfo.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpack/src/main/resources/common-services/MICROSOFT_R/8.0.0/configuration/microsoft-r-env.xml
>  PRE-CREATION 
>   
> contrib/management-packs/microsoft-r_mpa

Re: Review Request 51844: Audit logs should not be filtered by Log Feeder

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51844/#review148774
---


Ship it!




Ship It!

- Oliver Szabo


On Sept. 13, 2016, 12:37 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51844/
> ---
> 
> (Updated Sept. 13, 2016, 12:37 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18372
> https://issues.apache.org/jira/browse/AMBARI-18372
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> LogFeeder shouldn't filter audit logs.
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/FilterLogData.java
>  801a289 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
>  2c81c19 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandlerTest.java
>  02ffd47 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputManagerTest.java
>  e103346 
> 
> Diff: https://reviews.apache.org/r/51844/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 51540: Expose Disabling of Alert Targets in Web Client

[Zhe (Joe) Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51540/#review148775
---


Ship it!




Ship It!

- Zhe (Joe) Wang


On Sept. 9, 2016, 10:07 p.m., Vivek Ratnavel Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51540/
> ---
> 
> (Updated Sept. 9, 2016, 10:07 p.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly, Zhe (Joe) Wang, and Yusaku Sako.
> 
> 
> Bugs: Ambari-18281
> https://issues.apache.org/jira/browse/Ambari-18281
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add an option to Enable/ Disable alert notifications in Manage Notifications 
> popup.
> 
> 
> Diffs
> -
> 
>   
> ambari-web/app/controllers/main/alerts/alert_definitions_actions_controller.js
>  f3f0387 
>   ambari-web/app/styles/modal_popups.less a2343e0 
> 
> Diff: https://reviews.apache.org/r/51540/diff/
> 
> 
> Testing
> ---
> 
> Verified manually.
> Ambari-web unit tests pass.
> 30014 tests complete (26 seconds)
> 154 tests pending
> 
> 
> File Attachments
> 
> 
> v1
>   
> https://reviews.apache.org/media/uploaded/files/2016/08/31/ad971a3a-3430-4caa-9e7f-df60cc75872c__AMBARI-18281.v1.patch
> 
> 
> Thanks,
> 
> Vivek Ratnavel Subramanian
> 
>

Re: Review Request 51859: Enable Kerberos Wizard: Metrics collector fails to start on a cluster with Ranger installed

[Zhe (Joe) Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51859/#review148777
---


Ship it!




Ship It!

- Zhe (Joe) Wang


On Sept. 13, 2016, 8:40 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51859/
> ---
> 
> (Updated Sept. 13, 2016, 8:40 p.m.)
> 
> 
> Review request for Ambari, Zhe (Joe) Wang, Richard Zang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-18382
> https://issues.apache.org/jira/browse/AMBARI-18382
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This happens due to incorrect config property update of Ambari metrics service
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/admin/kerberos/step4_controller.js 91a40dd 
> 
> Diff: https://reviews.apache.org/r/51859/diff/
> 
> 
> Testing
> ---
> 
> Verified that patch fixes the issue on a cluster.
> Verified that all ambari-web unit tests passes:
> 29244 tests complete (29 seconds)
> 154 tests pending
> 
> 
> Thanks,
> 
> Jaimin Jetly
> 
>

Review Request 51859: Enable Kerberos Wizard: Metrics collector fails to start on a cluster with Ranger installed

[Jaimin Jetly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51859/
---

Review request for Ambari, Zhe (Joe) Wang, Richard Zang, and Yusaku Sako.


Bugs: AMBARI-18382
https://issues.apache.org/jira/browse/AMBARI-18382


Repository: ambari


Description
---

This happens due to incorrect config property update of Ambari metrics service


Diffs
-

  ambari-web/app/controllers/main/admin/kerberos/step4_controller.js 91a40dd 

Diff: https://reviews.apache.org/r/51859/diff/


Testing
---

Verified that patch fixes the issue on a cluster.
Verified that all ambari-web unit tests passes:
29244 tests complete (29 seconds)
154 tests pending


Thanks,

Jaimin Jetly

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/
---

(Updated Sept. 13, 2016, 4:44 p.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.


Changes
---

Addressed (some) reviewer concerns.


Bugs: AMBARI-18365
https://issues.apache.org/jira/browse/AMBARI-18365


Repository: ambari


Description
---

Add the followng Ambari configuration options to support Kerberos token 
authentication

- `authentication.kerberos.enabled`
-- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
Ambari:  {{true}} to enable this feature; {{false}}, otherwise
- `authentication.kerberos.spnego.principal`
-- The Kerberos principal name to use when verifying user-supplied Kerberos 
tokens for authentication via SPNEGO
- `authentication.kerberos.spnego.keytab.file`
-- The Kerberos keytab file to use when verifying user-supplied Kerberos tokens 
for authentication via SPNEGO
- `authentication.kerberos.user.types`
-- A comma-delimited (ordered) list of preferred user types to use when finding 
the Ambari user account for the user-supplied Kerberos identity during 
authentication via SPNEGO
- `authentication.kerberos.auth_to_local.rules`
-- The auth-to-local rules set to use when translating a user's principal name 
to a local user name during authentication via SPNEGO.

NOTE: These properties are in the {{ambari.properties}} file since this feature 
may be enabled whether the rest of the cluster has Kerberos enabled or not. 

See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
requiring the need for this patch.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 ee73b8d 
  
ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
 f429a36 
  
ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/51822/diff/


Testing
---

Manually tested...

# Local test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 2:51:23.081s
[INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
[INFO] Final Memory: 48M/1792M
[INFO] 

# Jenkins test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 01:34 h
[INFO] Finished at: 2016-09-13T08:46:23+00:00
[INFO] Final Memory: 178M/2672M
[INFO] 

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12828101/AMBARI-18365_trunk_01.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
ambari-server.

Test results: 
https://builds.apache.org/job/Ambari-trunk-test-patch/8643//testReport/
Console output: 
https://builds.apache.org/job/Ambari-trunk-test-patch/8643//console

This message is automatically generated.


Thanks,

Robert Levas

Re: Review Request 51859: Enable Kerberos Wizard: Metrics collector fails to start on a cluster with Ranger installed

[Yusaku Sako]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51859/#review148778
---


Ship it!




Ship It!

- Yusaku Sako


On Sept. 13, 2016, 8:40 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51859/
> ---
> 
> (Updated Sept. 13, 2016, 8:40 p.m.)
> 
> 
> Review request for Ambari, Zhe (Joe) Wang, Richard Zang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-18382
> https://issues.apache.org/jira/browse/AMBARI-18382
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This happens due to incorrect config property update of Ambari metrics service
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/admin/kerberos/step4_controller.js 91a40dd 
> 
> Diff: https://reviews.apache.org/r/51859/diff/
> 
> 
> Testing
> ---
> 
> Verified that patch fixes the issue on a cluster.
> Verified that all ambari-web unit tests passes:
> 29244 tests complete (29 seconds)
> 154 tests pending
> 
> 
> Thanks,
> 
> Jaimin Jetly
> 
>

Re: Review Request 51859: Enable Kerberos Wizard: Metrics collector fails to start on a cluster with Ranger installed

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51859/#review148779
---


Ship it!




Ship It!

- Sumit Mohanty


On Sept. 13, 2016, 8:40 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51859/
> ---
> 
> (Updated Sept. 13, 2016, 8:40 p.m.)
> 
> 
> Review request for Ambari, Zhe (Joe) Wang, Richard Zang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-18382
> https://issues.apache.org/jira/browse/AMBARI-18382
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This happens due to incorrect config property update of Ambari metrics service
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/admin/kerberos/step4_controller.js 91a40dd 
> 
> Diff: https://reviews.apache.org/r/51859/diff/
> 
> 
> Testing
> ---
> 
> Verified that patch fixes the issue on a cluster.
> Verified that all ambari-web unit tests passes:
> 29244 tests complete (29 seconds)
> 154 tests pending
> 
> 
> Thanks,
> 
> Jaimin Jetly
> 
>

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Jonathan Hurley]

> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 5292-5294
> > 
> >
> > Why not just return here which would prevent you from needing to embed 
> > the rest of the logic in the if-statement
> 
> Robert Levas wrote:
> Then I would have 2 return statements... but I can make the chance since 
> most developers like to short-curcuit their code.

That's true; it's 50/50 on what preferences are for this stuff. I find that 
with Ambari, we have a lot of nested if-statements, so we end up with code like:
```
if(null != cluster)
  if(null != cluster.getServices())
if(null != service.getName())
  // do something
  
```
Where the something is quite complex and has it's own control flow. Because of 
this, I think it's easier and less error prone to quickly return for nulls.


> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  lines 5298-5301
> > 
> >
> > KERBEROS_AUTH_USER_TYPES has a default value, so `getProperty` will 
> > never return null.
> 
> Robert Levas wrote:
> This is a safety measure in the event someone changes the definition of 
> `KERBEROS_AUTH_USER_TYPES`. Plus I generally feel the need to check for null 
> and empty in these cases. But I can remove this check.

Nope, that's fine - you can keep it then - I was just pointing out that you're 
getting the default for free here so unless somebody sets it to blank in 
`ambari.properties` it should never be null.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148699
---


On Sept. 13, 2016, 4:44 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 4:44 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:51:23.081s
> [INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
> [INFO] Final Memory: 48M/1792M
> [INFO] 
> 
> 
> # Jenkins test results: 
> 
> [INF

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Jonathan Hurley]

> On Sept. 13, 2016, 8:24 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java,
> >  line 2586
> > 
> >
> > Although I understand the motivation, I still don't like doing this 
> > since getting the backing Properties object for the entire Configuration 
> > would be missing these properties. Can we mirror them in the normal 
> > Properties object as well?
> 
> Robert Levas wrote:
> I am not sure what you mean.  The backing properties object contains the 
> properties from the ambari.properties file and is being used to fill the 
> KerberosAuthenticationProperties instance.   Does this data need to be copied 
> to another map?

Yeah, I missed that it was coming from the backing properties. Sorry.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148699
---


On Sept. 13, 2016, 4:44 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 4:44 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:51:23.081s
> [INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
> [INFO] Final Memory: 48M/1792M
> [INFO] 
> 
> 
> # Jenkins test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 01:34 h
> [INFO] Finished at: 2016-09-13T08:46:23+00:00
> [INFO] Final Memory: 178M/2672M
> [INFO] 
> 
> 
> {color:green}+1 overall{color}.  Here are the results of testing the latest 
> attachment 
>   
> http://issues.apache.org/jira/secure/attachment/12828101/AMBARI-18365_trunk_01.patch
>   against trunk revision .
> 
> {color:green}+1 @author{color}.  The patch does not contain any @author 
> tags.
> 
> {color:green}+1 tests included{color}.  The patch appears to include 2 
> new or modified test files.
> 
> {color:green}+1 javac{color}.  The applied patch does not increase the 
> total number of javac compiler wa

Re: Review Request 51822: Add Ambari configuration options to support Kerberos token authentication

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51822/#review148785
---


Ship it!




Ship It!

- Jonathan Hurley


On Sept. 13, 2016, 4:44 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51822/
> ---
> 
> (Updated Sept. 13, 2016, 4:44 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-18365
> https://issues.apache.org/jira/browse/AMBARI-18365
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add the followng Ambari configuration options to support Kerberos token 
> authentication
> 
> - `authentication.kerberos.enabled`
> -- Determines whether to use Kerberos (SPNEGO) authentication when connecting 
> Ambari:  {{true}} to enable this feature; {{false}}, otherwise
> - `authentication.kerberos.spnego.principal`
> -- The Kerberos principal name to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.spnego.keytab.file`
> -- The Kerberos keytab file to use when verifying user-supplied Kerberos 
> tokens for authentication via SPNEGO
> - `authentication.kerberos.user.types`
> -- A comma-delimited (ordered) list of preferred user types to use when 
> finding the Ambari user account for the user-supplied Kerberos identity 
> during authentication via SPNEGO
> - `authentication.kerberos.auth_to_local.rules`
> -- The auth-to-local rules set to use when translating a user's principal 
> name to a local user name during authentication via SPNEGO.
> 
> NOTE: These properties are in the {{ambari.properties}} file since this 
> feature may be enabled whether the rest of the cluster has Kerberos enabled 
> or not. 
> 
> See https://issues.apache.org/jira/browse/AMBARI-18364 for an overview 
> requiring the need for this patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  ee73b8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java
>  f429a36 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationPropertiesTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51822/diff/
> 
> 
> Testing
> ---
> 
> Manually tested...
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:51:23.081s
> [INFO] Finished at: Tue Sep 13 00:00:59 EDT 2016
> [INFO] Final Memory: 48M/1792M
> [INFO] 
> 
> 
> # Jenkins test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 01:34 h
> [INFO] Finished at: 2016-09-13T08:46:23+00:00
> [INFO] Final Memory: 178M/2672M
> [INFO] 
> 
> 
> {color:green}+1 overall{color}.  Here are the results of testing the latest 
> attachment 
>   
> http://issues.apache.org/jira/secure/attachment/12828101/AMBARI-18365_trunk_01.patch
>   against trunk revision .
> 
> {color:green}+1 @author{color}.  The patch does not contain any @author 
> tags.
> 
> {color:green}+1 tests included{color}.  The patch appears to include 2 
> new or modified test files.
> 
> {color:green}+1 javac{color}.  The applied patch does not increase the 
> total number of javac compiler warnings.
> 
> {color:green}+1 release audit{color}.  The applied patch does not 
> increase the total number of release audit warnings.
> 
> {color:green}+1 core tests{color}.  The patch passed unit tests in 
> ambari-server.
> 
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/8643//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/8643//console
> 
> This message is automatically generated.
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 51815: Introduce conditional dependencies in stack defition to handle blueprint validation gracefully

[Amruta Borkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51815/
---

(Updated Sept. 13, 2016, 9:28 p.m.)


Review request for Ambari, Shantanu Mundkur, Di Li, Juanjo  Marron, Laszlo 
Puskas, and Robert Nettleton.


Bugs: AMBARI-18355
https://issues.apache.org/jira/browse/AMBARI-18355


Repository: ambari


Description
---

Currently stack definitions do not list conditional dependencies, adding those 
to the stack definitions would make it easy to validate errors in case of 
blueprint deployment. Please refer to document attached to Jira


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/state/DependencyConditionInfo.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/DependencyInfo.java 
e3db662 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
 a5f33ff 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
65d166a 
  
ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java
 b1de8ef 

Diff: https://reviews.apache.org/r/51815/diff/


Testing
---

Written Junit test cases. Perfomred manual testing to check the Namenode HA 
component dependency. Was able to proceed with the installation for valid 
blueprint. and got validation error message while registering blueprint when 
the Blueprint did not satisfy the conditional dependencies.


Thanks,

Amruta Borkar

Re: Review Request 51815: Introduce conditional dependencies in stack defition to handle blueprint validation gracefully

[Amruta Borkar]

> On Sept. 13, 2016, 2:07 p.m., Di Li wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/DependencyConditionInfo.java,
> >  line 103
> > 
> >
> > do you support other type of checks ? such as the property has to be a 
> > certain value ?

Hi Di,
Yes, property value check is handled by the tag  
 as given in the document. This tag is optional and if it is 
present, then the blueprint will be validated for the the  
mentioned in metainfo.xml file. 
This check is performed by following code in isResolved() method:
//if 'propertyValue' is null then it is assumed that condition only 
checks if the 'property' exists or not
if(propertyValue == null || 
propertyValue.equals(properties.get(configType).get(property))) return true;

For this scenario (NAMENODE HA) we do not have conditional dependency based on 
propertyValue hence I have not added it in the metainfo.xml file.


> On Sept. 13, 2016, 2:07 p.m., Di Li wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml,
> >  line 58
> > 
> >
> > what does "dfs.nameservices" mean in this 
> > conditional check? as a negative test, what if a user manually added the 
> > property to hdfs-site.xml ?

When a strucutre like  
   .. 
   ... 
   
is present in metainfo.xml file code will look to see if this property is 
present in blueprint. If not then the  will not be considered 
mandatory. 

I have modified the patch to consider configurations provided in hdfs-site.xml.


> On Sept. 13, 2016, 2:07 p.m., Di Li wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml,
> >  line 55
> > 
> >
> > this is already a subsection of "dependency", it can just be called 
> > "conditions"

Modified the patch.


> On Sept. 13, 2016, 2:07 p.m., Di Li wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml,
> >  line 56
> > 
> >
> > same here, no need to reiterate it's for "dependency"

Modified the patch.


- Amruta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51815/#review148721
---


On Sept. 13, 2016, 9:28 p.m., Amruta Borkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51815/
> ---
> 
> (Updated Sept. 13, 2016, 9:28 p.m.)
> 
> 
> Review request for Ambari, Shantanu Mundkur, Di Li, Juanjo  Marron, Laszlo 
> Puskas, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-18355
> https://issues.apache.org/jira/browse/AMBARI-18355
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently stack definitions do not list conditional dependencies, adding 
> those to the stack definitions would make it easy to validate errors in case 
> of blueprint deployment. Please refer to document attached to Jira
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/DependencyConditionInfo.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/DependencyInfo.java
>  e3db662 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
>  a5f33ff 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> 65d166a 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java
>  b1de8ef 
> 
> Diff: https://reviews.apache.org/r/51815/diff/
> 
> 
> Testing
> ---
> 
> Written Junit test cases. Perfomred manual testing to check the Namenode HA 
> component dependency. Was able to proceed with the installation for valid 
> blueprint. and got validation error message while registering blueprint when 
> the Blueprint did not satisfy the conditional dependencies.
> 
> 
> Thanks,
> 
> Amruta Borkar
> 
>

Review Request 51863: AMBARI-18383 : Oozie spark failed due to missing py4j and/or pyspark zip files in share lib

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51863/
---

Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Sumit Mohanty, and 
Venkat Ranganathan.


Bugs: AMBARI-18383
https://issues.apache.org/jira/browse/AMBARI-18383


Repository: ambari


Description
---

Failed exception:

"Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], 
main() 
threw exception, Missing py4j and/or pyspark zip files. Please add them to the 
lib folder 
or to the Spark sharelib.

org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j 
and/or 
pyspark zip files. Please add them to the lib folder or to the Spark sharelib."


Those two files were not present in the share lib
hrt_qa@hn0-3ad6fe:~$ hdfs dfs -ls /user/oozie/share/lib/spark/*.zip
ls: `/user/oozie/share/lib/spark/*.zip': No such file or directory


Diffs
-

  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
 8d80638 
  ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 

Diff: https://reviews.apache.org/r/51863/diff/


Testing
---

Manually tested on sys-prepped cluster. 

Execution log

2016-09-13 21:17:32,590 - Execute[('mv', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig')] {'sudo': True}
2016-09-13 21:17:32,613 - 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'owner': 'oozie', 
'create_parents': True}
2016-09-13 21:17:32,635 - Creating directory 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] since it doesn't exist.
2016-09-13 21:17:32,746 - Changing owner for 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark from 0 to oozie
2016-09-13 21:17:32,771 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig/oozie-sharelib-spark*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:32,804 - Execute['cp -P /usr/hdp/2.5.1.0-16/spark/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,125 - Execute['find 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/ -type l -delete'] {'user': 'oozie'}
2016-09-13 21:17:33,163 - Execute['rm -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/spark-examples*.jar'] {'user': 
'oozie'}
2016-09-13 21:17:33,220 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.zip 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,258 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
No jar files found in Spark client python lib.
2016-09-13 21:17:33,293 - HdfsResource['/tmp/oozie/user/oozie/share'] 
{'security_enabled': False, 'hadoop_conf_dir': '/etc/hadoop/conf', 'keytab': 
None, 'source': '/usr/hdp/2.5.1.0-16/oozie/share', 'default_fs': 
'adl://oozie3hadl25.azuredatalakestore.net:443', 'user': 'hdfs', 
'hdfs_resource_ignore_file': 
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': 
{'dfs.webhdfs.enabled': False}, 'kinit_path_local': None, 'principal_name': 
None, 'recursive_chmod': True, 'owner': 'oozie', 'hadoop_bin_dir': 
'/usr/hdp/2.5.1.0-16/hadoop/bin', 'type': 'directory', 'action': 
['create_on_execute'], 'mode': 0755}


Thanks,

Aravindan Vijayan

Re: Review Request 51863: AMBARI-18383 : Oozie spark failed due to incomplete spark share-lib directory

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51863/
---

(Updated Sept. 13, 2016, 9:36 p.m.)


Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Sumit Mohanty, and 
Venkat Ranganathan.


Summary (updated)
-

AMBARI-18383 : Oozie spark failed due to incomplete spark share-lib directory


Bugs: AMBARI-18383
https://issues.apache.org/jira/browse/AMBARI-18383


Repository: ambari


Description
---

Failed exception:

"Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], 
main() 
threw exception, Missing py4j and/or pyspark zip files. Please add them to the 
lib folder 
or to the Spark sharelib.

org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j 
and/or 
pyspark zip files. Please add them to the lib folder or to the Spark sharelib."


Those two files were not present in the share lib
hrt_qa@hn0-3ad6fe:~$ hdfs dfs -ls /user/oozie/share/lib/spark/*.zip
ls: `/user/oozie/share/lib/spark/*.zip': No such file or directory


Diffs
-

  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
 8d80638 
  ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 

Diff: https://reviews.apache.org/r/51863/diff/


Testing
---

Manually tested on sys-prepped cluster. 

Execution log

2016-09-13 21:17:32,590 - Execute[('mv', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig')] {'sudo': True}
2016-09-13 21:17:32,613 - 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'owner': 'oozie', 
'create_parents': True}
2016-09-13 21:17:32,635 - Creating directory 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] since it doesn't exist.
2016-09-13 21:17:32,746 - Changing owner for 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark from 0 to oozie
2016-09-13 21:17:32,771 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig/oozie-sharelib-spark*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:32,804 - Execute['cp -P /usr/hdp/2.5.1.0-16/spark/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,125 - Execute['find 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/ -type l -delete'] {'user': 'oozie'}
2016-09-13 21:17:33,163 - Execute['rm -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/spark-examples*.jar'] {'user': 
'oozie'}
2016-09-13 21:17:33,220 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.zip 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,258 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
No jar files found in Spark client python lib.
2016-09-13 21:17:33,293 - HdfsResource['/tmp/oozie/user/oozie/share'] 
{'security_enabled': False, 'hadoop_conf_dir': '/etc/hadoop/conf', 'keytab': 
None, 'source': '/usr/hdp/2.5.1.0-16/oozie/share', 'default_fs': 
'adl://oozie3hadl25.azuredatalakestore.net:443', 'user': 'hdfs', 
'hdfs_resource_ignore_file': 
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': 
{'dfs.webhdfs.enabled': False}, 'kinit_path_local': None, 'principal_name': 
None, 'recursive_chmod': True, 'owner': 'oozie', 'hadoop_bin_dir': 
'/usr/hdp/2.5.1.0-16/hadoop/bin', 'type': 'directory', 'action': 
['create_on_execute'], 'mode': 0755}


Thanks,

Aravindan Vijayan

Re: Review Request 51863: AMBARI-18383 : Oozie spark failed due to incomplete spark share-lib directory

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51863/#review148790
---




ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
 (line 126)


Typically, we need a 
params.HdfsResource(None, action="execute")
after HdfsResource(...) calls. That way, code accumulates all HDFS calls 
and executes them one shot. In this case, I think we need to add the line after 
the call.


- Sumit Mohanty


On Sept. 13, 2016, 9:36 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51863/
> ---
> 
> (Updated Sept. 13, 2016, 9:36 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Sumit Mohanty, and 
> Venkat Ranganathan.
> 
> 
> Bugs: AMBARI-18383
> https://issues.apache.org/jira/browse/AMBARI-18383
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Failed exception:
> 
> "Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() 
> threw exception, Missing py4j and/or pyspark zip files. Please add them to 
> the lib folder 
> or to the Spark sharelib.
> 
> org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j 
> and/or 
> pyspark zip files. Please add them to the lib folder or to the Spark 
> sharelib."
> 
> 
> Those two files were not present in the share lib
> hrt_qa@hn0-3ad6fe:~$ hdfs dfs -ls /user/oozie/share/lib/spark/*.zip
> ls: `/user/oozie/share/lib/spark/*.zip': No such file or directory
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
>  8d80638 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
> 
> Diff: https://reviews.apache.org/r/51863/diff/
> 
> 
> Testing
> ---
> 
> Manually tested on sys-prepped cluster. 
> 
> Execution log
> 
> 2016-09-13 21:17:32,590 - Execute[('mv', 
> '/usr/hdp/2.5.1.0-16/oozie/share/lib/spark', 
> '/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig')] {'sudo': True}
> 2016-09-13 21:17:32,613 - 
> Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'owner': 'oozie', 
> 'create_parents': True}
> 2016-09-13 21:17:32,635 - Creating directory 
> Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] since it doesn't exist.
> 2016-09-13 21:17:32,746 - Changing owner for 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark from 0 to oozie
> 2016-09-13 21:17:32,771 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig/oozie-sharelib-spark*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:32,804 - Execute['cp -P /usr/hdp/2.5.1.0-16/spark/lib/*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:33,125 - Execute['find 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark/ -type l -delete'] {'user': 'oozie'}
> 2016-09-13 21:17:33,163 - Execute['rm -f 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark/spark-examples*.jar'] {'user': 
> 'oozie'}
> 2016-09-13 21:17:33,220 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/spark/python/lib/*.zip 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:33,258 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/spark/python/lib/*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> No jar files found in Spark client python lib.
> 2016-09-13 21:17:33,293 - HdfsResource['/tmp/oozie/user/oozie/share'] 
> {'security_enabled': False, 'hadoop_conf_dir': '/etc/hadoop/conf', 'keytab': 
> None, 'source': '/usr/hdp/2.5.1.0-16/oozie/share', 'default_fs': 
> 'adl://oozie3hadl25.azuredatalakestore.net:443', 'user': 'hdfs', 
> 'hdfs_resource_ignore_file': 
> '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': 
> {'dfs.webhdfs.enabled': False}, 'kinit_path_local': None, 'principal_name': 
> None, 'recursive_chmod': True, 'owner': 'oozie', 'hadoop_bin_dir': 
> '/usr/hdp/2.5.1.0-16/hadoop/bin', 'type': 'directory', 'action': 
> ['create_on_execute'], 'mode': 0755}
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 51863: AMBARI-18383 : Oozie spark failed due to incomplete spark share-lib directory

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51863/
---

(Updated Sept. 13, 2016, 9:56 p.m.)


Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Sumit Mohanty, and 
Venkat Ranganathan.


Bugs: AMBARI-18383
https://issues.apache.org/jira/browse/AMBARI-18383


Repository: ambari


Description
---

Failed exception:

"Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], 
main() 
threw exception, Missing py4j and/or pyspark zip files. Please add them to the 
lib folder 
or to the Spark sharelib.

org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j 
and/or 
pyspark zip files. Please add them to the lib folder or to the Spark sharelib."


Those two files were not present in the share lib
hrt_qa@hn0-3ad6fe:~$ hdfs dfs -ls /user/oozie/share/lib/spark/*.zip
ls: `/user/oozie/share/lib/spark/*.zip': No such file or directory


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
 ee97a89 
  ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 

Diff: https://reviews.apache.org/r/51863/diff/


Testing
---

Manually tested on sys-prepped cluster. 

Execution log

2016-09-13 21:17:32,590 - Execute[('mv', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig')] {'sudo': True}
2016-09-13 21:17:32,613 - 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'owner': 'oozie', 
'create_parents': True}
2016-09-13 21:17:32,635 - Creating directory 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] since it doesn't exist.
2016-09-13 21:17:32,746 - Changing owner for 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark from 0 to oozie
2016-09-13 21:17:32,771 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig/oozie-sharelib-spark*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:32,804 - Execute['cp -P /usr/hdp/2.5.1.0-16/spark/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,125 - Execute['find 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/ -type l -delete'] {'user': 'oozie'}
2016-09-13 21:17:33,163 - Execute['rm -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/spark-examples*.jar'] {'user': 
'oozie'}
2016-09-13 21:17:33,220 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.zip 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,258 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
No jar files found in Spark client python lib.
2016-09-13 21:17:33,293 - HdfsResource['/tmp/oozie/user/oozie/share'] 
{'security_enabled': False, 'hadoop_conf_dir': '/etc/hadoop/conf', 'keytab': 
None, 'source': '/usr/hdp/2.5.1.0-16/oozie/share', 'default_fs': 
'adl://oozie3hadl25.azuredatalakestore.net:443', 'user': 'hdfs', 
'hdfs_resource_ignore_file': 
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': 
{'dfs.webhdfs.enabled': False}, 'kinit_path_local': None, 'principal_name': 
None, 'recursive_chmod': True, 'owner': 'oozie', 'hadoop_bin_dir': 
'/usr/hdp/2.5.1.0-16/hadoop/bin', 'type': 'directory', 'action': 
['create_on_execute'], 'mode': 0755}


Thanks,

Aravindan Vijayan

Re: Review Request 51833: Atlas web UI alert after performing stack upgrade to HDP 2.5 and adding Atlas Service

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51833/
---

(Updated Sept. 13, 2016, 10:11 p.m.)


Review request for Ambari, Sumit Mohanty and Swapan Shridhar.


Bugs: AMBARI-18368
https://issues.apache.org/jira/browse/AMBARI-18368


Repository: ambari


Description
---

Steps to Reproduce:
* Install Ambari 2.2.2 with HDP 2.4 (HBase, Solr)
* Kerberize the cluster
* Perform EU/RU to HDP 2.5
* Add Atlas Service

Atlas Server log contains,
```
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at 
http://natu146-ehbs-dgm10toeriesec-u14-1.openstacklocal:8886/solr: Can not find 
the specified config set: vertex_index  
```

Fix was to change ownership of users-credentials.properties and 
policy-store.txt to atlas:hadoop


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 0c07018 

Diff: https://reviews.apache.org/r/51833/diff/


Testing
---

Verified on a live cluster.

Python unit tests passed.
Total run:1124
Total errors:0
Total failures:0
OK


Thanks,

Alejandro Fernandez

Re: Review Request 51833: Atlas web UI alert after performing stack upgrade to HDP 2.5 and adding Atlas Service

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51833/
---

(Updated Sept. 13, 2016, 10:32 p.m.)


Review request for Ambari, Sumit Mohanty and Swapan Shridhar.


Bugs: AMBARI-18368
https://issues.apache.org/jira/browse/AMBARI-18368


Repository: ambari


Description
---

Steps to Reproduce:
* Install Ambari 2.2.2 with HDP 2.4 (HBase, Solr)
* Kerberize the cluster
* Perform EU/RU to HDP 2.5
* Add Atlas Service

Atlas Server log contains,
```
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at 
http://natu146-ehbs-dgm10toeriesec-u14-1.openstacklocal:8886/solr: Can not find 
the specified config set: vertex_index  
```

Fix was to change ownership of users-credentials.properties and 
policy-store.txt to atlas:hadoop


Diffs
-

  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 0c07018 

Diff: https://reviews.apache.org/r/51833/diff/


Testing
---

Verified on a live cluster.

Python unit tests passed.
Total run:1124
Total errors:0
Total failures:0
OK


Thanks,

Alejandro Fernandez

Re: Review Request 51863: AMBARI-18383 : Oozie spark failed due to incomplete spark share-lib directory

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51863/#review148798
---


Ship it!




Ship It!

- Sumit Mohanty


On Sept. 13, 2016, 9:56 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51863/
> ---
> 
> (Updated Sept. 13, 2016, 9:56 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Sumit Mohanty, and 
> Venkat Ranganathan.
> 
> 
> Bugs: AMBARI-18383
> https://issues.apache.org/jira/browse/AMBARI-18383
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Failed exception:
> 
> "Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() 
> threw exception, Missing py4j and/or pyspark zip files. Please add them to 
> the lib folder 
> or to the Spark sharelib.
> 
> org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j 
> and/or 
> pyspark zip files. Please add them to the lib folder or to the Spark 
> sharelib."
> 
> 
> Those two files were not present in the share lib
> hrt_qa@hn0-3ad6fe:~$ hdfs dfs -ls /user/oozie/share/lib/spark/*.zip
> ls: `/user/oozie/share/lib/spark/*.zip': No such file or directory
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
>  ee97a89 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
> 
> Diff: https://reviews.apache.org/r/51863/diff/
> 
> 
> Testing
> ---
> 
> Manually tested on sys-prepped cluster. 
> 
> Execution log
> 
> 2016-09-13 21:17:32,590 - Execute[('mv', 
> '/usr/hdp/2.5.1.0-16/oozie/share/lib/spark', 
> '/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig')] {'sudo': True}
> 2016-09-13 21:17:32,613 - 
> Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'owner': 'oozie', 
> 'create_parents': True}
> 2016-09-13 21:17:32,635 - Creating directory 
> Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] since it doesn't exist.
> 2016-09-13 21:17:32,746 - Changing owner for 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark from 0 to oozie
> 2016-09-13 21:17:32,771 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig/oozie-sharelib-spark*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:32,804 - Execute['cp -P /usr/hdp/2.5.1.0-16/spark/lib/*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:33,125 - Execute['find 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark/ -type l -delete'] {'user': 'oozie'}
> 2016-09-13 21:17:33,163 - Execute['rm -f 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark/spark-examples*.jar'] {'user': 
> 'oozie'}
> 2016-09-13 21:17:33,220 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/spark/python/lib/*.zip 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:33,258 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/spark/python/lib/*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> No jar files found in Spark client python lib.
> 2016-09-13 21:17:33,293 - HdfsResource['/tmp/oozie/user/oozie/share'] 
> {'security_enabled': False, 'hadoop_conf_dir': '/etc/hadoop/conf', 'keytab': 
> None, 'source': '/usr/hdp/2.5.1.0-16/oozie/share', 'default_fs': 
> 'adl://oozie3hadl25.azuredatalakestore.net:443', 'user': 'hdfs', 
> 'hdfs_resource_ignore_file': 
> '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': 
> {'dfs.webhdfs.enabled': False}, 'kinit_path_local': None, 'principal_name': 
> None, 'recursive_chmod': True, 'owner': 'oozie', 'hadoop_bin_dir': 
> '/usr/hdp/2.5.1.0-16/hadoop/bin', 'type': 'directory', 'action': 
> ['create_on_execute'], 'mode': 0755}
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 51863: AMBARI-18383 : Oozie spark failed due to incomplete spark share-lib directory

[Venkat Ranganathan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51863/#review148807
---


Ship it!




Ship It!

- Venkat Ranganathan


On Sept. 13, 2016, 2:56 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51863/
> ---
> 
> (Updated Sept. 13, 2016, 2:56 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Sumit Mohanty, and 
> Venkat Ranganathan.
> 
> 
> Bugs: AMBARI-18383
> https://issues.apache.org/jira/browse/AMBARI-18383
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Failed exception:
> 
> "Failing Oozie Launcher, Main class 
> [org.apache.oozie.action.hadoop.SparkMain], main() 
> threw exception, Missing py4j and/or pyspark zip files. Please add them to 
> the lib folder 
> or to the Spark sharelib.
> 
> org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j 
> and/or 
> pyspark zip files. Please add them to the lib folder or to the Spark 
> sharelib."
> 
> 
> Those two files were not present in the share lib
> hrt_qa@hn0-3ad6fe:~$ hdfs dfs -ls /user/oozie/share/lib/spark/*.zip
> ls: `/user/oozie/share/lib/spark/*.zip': No such file or directory
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
>  ee97a89 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 
> 
> Diff: https://reviews.apache.org/r/51863/diff/
> 
> 
> Testing
> ---
> 
> Manually tested on sys-prepped cluster. 
> 
> Execution log
> 
> 2016-09-13 21:17:32,590 - Execute[('mv', 
> '/usr/hdp/2.5.1.0-16/oozie/share/lib/spark', 
> '/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig')] {'sudo': True}
> 2016-09-13 21:17:32,613 - 
> Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'owner': 'oozie', 
> 'create_parents': True}
> 2016-09-13 21:17:32,635 - Creating directory 
> Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] since it doesn't exist.
> 2016-09-13 21:17:32,746 - Changing owner for 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark from 0 to oozie
> 2016-09-13 21:17:32,771 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig/oozie-sharelib-spark*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:32,804 - Execute['cp -P /usr/hdp/2.5.1.0-16/spark/lib/*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:33,125 - Execute['find 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark/ -type l -delete'] {'user': 'oozie'}
> 2016-09-13 21:17:33,163 - Execute['rm -f 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark/spark-examples*.jar'] {'user': 
> 'oozie'}
> 2016-09-13 21:17:33,220 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/spark/python/lib/*.zip 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> 2016-09-13 21:17:33,258 - Execute['cp -f 
> /usr/hdp/2.5.1.0-16/spark/python/lib/*.jar 
> /usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
> No jar files found in Spark client python lib.
> 2016-09-13 21:17:33,293 - HdfsResource['/tmp/oozie/user/oozie/share'] 
> {'security_enabled': False, 'hadoop_conf_dir': '/etc/hadoop/conf', 'keytab': 
> None, 'source': '/usr/hdp/2.5.1.0-16/oozie/share', 'default_fs': 
> 'adl://oozie3hadl25.azuredatalakestore.net:443', 'user': 'hdfs', 
> 'hdfs_resource_ignore_file': 
> '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': 
> {'dfs.webhdfs.enabled': False}, 'kinit_path_local': None, 'principal_name': 
> None, 'recursive_chmod': True, 'owner': 'oozie', 'hadoop_bin_dir': 
> '/usr/hdp/2.5.1.0-16/hadoop/bin', 'type': 'directory', 'action': 
> ['create_on_execute'], 'mode': 0755}
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 51843: rolling restart datanode cluster name in audit log show null

[wang yaoxin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51843/
---

(Updated 九月 14, 2016, 6:04 a.m.)


Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sebastian 
Toader.


Bugs: AMBARI-18374
https://issues.apache.org/jira/browse/AMBARI-18374


Repository: ambari


Description
---

when rolling restart services ( eg:DataNode ) ,cluster name in audit log show 
null

2016-09-13T17:29:22.033+0800, User(admin), RemoteIp(127.0.0.1), 
Operation(Request from server), RequestType(POST), 
url(http://localhost:8080/api/v1/clusters/amabri/requests), ResultStatus(202 
Accepted), Command(RESTART), Cluster name(null)

2016-09-13T17:29:22.042+0800, User(admin), 
Operation(PARSE.ROLLING-RESTART.DATANODE.1.1), Status(IN_PROGRESS), 
RequestId(15)


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/audit/request/eventcreator/RequestEventCreator.java
 075e328 
  
ambari-server/src/test/java/org/apache/ambari/server/audit/request/creator/RequestEventCreatorTest.java
 1074f0e 

Diff: https://reviews.apache.org/r/51843/diff/


Testing
---

done!


Thanks,

wang yaoxin