Re: Review Request 63051: Add service group dependencies

2017-10-25 Thread Vitalyi Brodetskyi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63051/
---

(Updated Жов. 25, 2017, 11:18 після полудня)


Review request for Ambari, Jayush Luniya, Myroslav Papirkovskyy, and Swapan 
Shridhar.


Bugs: AMBARI-22249
https://issues.apache.org/jira/browse/AMBARI-22249


Repository: ambari


Description
---

Add table servicegroupdependencies


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
 737b9b4 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ServiceGroupDependencyResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ServiceGroupResourceDefinition.java
 13bdd7b 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ServiceGroupKey.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ServiceGroupService.java
 1e2eaf4 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 3117186 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ResourceProviderFactory.java
 f75a7d5 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ServiceGroupDependencyRequest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ServiceGroupDependencyResponse.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
 b8c1674 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/DefaultProviderModule.java
 cfab185 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceGroupDependencyResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceGroupResourceProvider.java
 2e935af 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java
 6a76fff 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ServiceGroupEntity.java
 7f8facb 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
8fd878e 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceGroup.java 
9850462 
  
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceGroupFactory.java
 7146bdf 
  
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceGroupImpl.java
 735a946 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 1e4d81f 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
 0434728 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql b8c0a42 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql c56e486 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 8917d48 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql f48bcd6 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql e7c5c72 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql f87af4c 


Diff: https://reviews.apache.org/r/63051/diff/8/

Changes: https://reviews.apache.org/r/63051/diff/7-8/


Testing
---

Will fix tests after patch approve


Thanks,

Vitalyi Brodetskyi

Re: Review Request 63305: Downgrade From HDP 2.6 to 2.5 Leaves 2.6 Hosts as CURRENT Instead of INSTALLED

2017-10-25 Thread Dmytro Grinenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63305/#review189242
---


Ship it!




Ship It!

- Dmytro Grinenko


On Oct. 25, 2017, 7:57 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63305/
> ---
> 
> (Updated Oct. 25, 2017, 7:57 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22312
> https://issues.apache.org/jira/browse/AMBARI-22312
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> - Install HDP 2.5.4 with ZK
> - Perform an RU to 2.6.0.0
> - Downgrade back to 2.5.4
> 
> All hosts remain current after finalization:
> {code}
> {
>   "href": 
> "http://localhost:8080/api/v1/clusters/c1/stack_versions?fields=ClusterStackVersions/host_states;,
>   "items": [
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/1;,
>   "ClusterStackVersions": {
> "cluster_name": "c1",
> "id": 1,
> "repository_version": 1,
> "stack": "HDP",
> "version": "2.5",
> "host_states": {
>   "CURRENT": [
> "c6403.ambari.apache.org",
> "c6402.ambari.apache.org",
> "c6401.ambari.apache.org"
>   ],
>   "INSTALLED": [],
>   "INSTALLING": [],
>   "INSTALL_FAILED": [],
>   "NOT_REQUIRED": [],
>   "OUT_OF_SYNC": []
> }
>   }
> },
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/2;,
>   "ClusterStackVersions": {
> "cluster_name": "c1",
> "id": 2,
> "repository_version": 2,
> "stack": "HDP",
> "version": "2.6",
> "host_states": {
>   "CURRENT": [
> "c6403.ambari.apache.org",
> "c6402.ambari.apache.org",
> "c6401.ambari.apache.org"
>   ],
>   "INSTALLED": [],
>   "INSTALLING": [],
>   "INSTALL_FAILED": [],
>   "NOT_REQUIRED": [],
>   "OUT_OF_SYNC": []
> }
>   }
> }
>   ]
> }
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  4d5f3ba627 
> 
> 
> Diff: https://reviews.apache.org/r/63305/diff/1/
> 
> 
> Testing
> ---
> 
> PENDING
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63305: Downgrade From HDP 2.6 to 2.5 Leaves 2.6 Hosts as CURRENT Instead of INSTALLED

2017-10-25 Thread Nate Cole 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63305/#review189241
---


Ship it!




Ship It!

- Nate Cole


On Oct. 25, 2017, 3:57 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63305/
> ---
> 
> (Updated Oct. 25, 2017, 3:57 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22312
> https://issues.apache.org/jira/browse/AMBARI-22312
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> - Install HDP 2.5.4 with ZK
> - Perform an RU to 2.6.0.0
> - Downgrade back to 2.5.4
> 
> All hosts remain current after finalization:
> {code}
> {
>   "href": 
> "http://localhost:8080/api/v1/clusters/c1/stack_versions?fields=ClusterStackVersions/host_states;,
>   "items": [
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/1;,
>   "ClusterStackVersions": {
> "cluster_name": "c1",
> "id": 1,
> "repository_version": 1,
> "stack": "HDP",
> "version": "2.5",
> "host_states": {
>   "CURRENT": [
> "c6403.ambari.apache.org",
> "c6402.ambari.apache.org",
> "c6401.ambari.apache.org"
>   ],
>   "INSTALLED": [],
>   "INSTALLING": [],
>   "INSTALL_FAILED": [],
>   "NOT_REQUIRED": [],
>   "OUT_OF_SYNC": []
> }
>   }
> },
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/2;,
>   "ClusterStackVersions": {
> "cluster_name": "c1",
> "id": 2,
> "repository_version": 2,
> "stack": "HDP",
> "version": "2.6",
> "host_states": {
>   "CURRENT": [
> "c6403.ambari.apache.org",
> "c6402.ambari.apache.org",
> "c6401.ambari.apache.org"
>   ],
>   "INSTALLED": [],
>   "INSTALLING": [],
>   "INSTALL_FAILED": [],
>   "NOT_REQUIRED": [],
>   "OUT_OF_SYNC": []
> }
>   }
> }
>   ]
> }
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  4d5f3ba627 
> 
> 
> Diff: https://reviews.apache.org/r/63305/diff/1/
> 
> 
> Testing
> ---
> 
> PENDING
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Review Request 63305: Downgrade From HDP 2.6 to 2.5 Leaves 2.6 Hosts as CURRENT Instead of INSTALLED

2017-10-25 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63305/
---

Review request for Ambari, Dmytro Grinenko and Nate Cole.


Bugs: AMBARI-22312
https://issues.apache.org/jira/browse/AMBARI-22312


Repository: ambari


Description
---

STR:
- Install HDP 2.5.4 with ZK
- Perform an RU to 2.6.0.0
- Downgrade back to 2.5.4

All hosts remain current after finalization:
{code}
{
  "href": 
"http://localhost:8080/api/v1/clusters/c1/stack_versions?fields=ClusterStackVersions/host_states;,
  "items": [
{
  "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/1;,
  "ClusterStackVersions": {
"cluster_name": "c1",
"id": 1,
"repository_version": 1,
"stack": "HDP",
"version": "2.5",
"host_states": {
  "CURRENT": [
"c6403.ambari.apache.org",
"c6402.ambari.apache.org",
"c6401.ambari.apache.org"
  ],
  "INSTALLED": [],
  "INSTALLING": [],
  "INSTALL_FAILED": [],
  "NOT_REQUIRED": [],
  "OUT_OF_SYNC": []
}
  }
},
{
  "href": "http://localhost:8080/api/v1/clusters/c1/stack_versions/2;,
  "ClusterStackVersions": {
"cluster_name": "c1",
"id": 2,
"repository_version": 2,
"stack": "HDP",
"version": "2.6",
"host_states": {
  "CURRENT": [
"c6403.ambari.apache.org",
"c6402.ambari.apache.org",
"c6401.ambari.apache.org"
  ],
  "INSTALLED": [],
  "INSTALLING": [],
  "INSTALL_FAILED": [],
  "NOT_REQUIRED": [],
  "OUT_OF_SYNC": []
}
  }
}
  ]
}
{code}


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
 4d5f3ba627 


Diff: https://reviews.apache.org/r/63305/diff/1/


Testing
---

PENDING


Thanks,

Jonathan Hurley

Re: Review Request 63205: AMBARI-22198. Implement read only API for getting stack level settings.

2017-10-25 Thread Swapan Shridhar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63205/
---

(Updated Oct. 25, 2017, 6:44 p.m.)


Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.


Changes
---

Added 
"ambari-server/src/main/resources/stacks/HDP/3.0/configuration/settings.xml" 
file.


Bugs: AMBARI-22198
https://issues.apache.org/jira/browse/AMBARI-22198


Repository: ambari


Description
---

This JIRA work assumes the following things to be present in the stack (3.0 
onwards) coming from the mpack (as there is no inheritance in stacks, given the 
flat stack structure coming from mpacks):

- **settings.xml** file in 
**/var/lib/ambari-server/resources/stacks///configuration**

- **stack_features.json**, **stack_packages.json** and **stack_tools.json** 
present in 
**/var/lib/ambari-server/resources/stacks///properties/**
 directory.


Above work is done as part of JIRA : AMBARI-22285


--


**Work done in this Review Request:**

- Implement the new read only 
http://:/api/v1/stacks//versions//settings 
API.
It serves the following settings : **stack_features**, **stack_name**, 
**stack_packages**, **stack_root** and **stack_tools**.

- Renaming of files for Cluster Settings related code (work done in  
AMBARI-22196).


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
 737b9b4 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/RootStackSettingsResourceDefinition.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 1c0a555 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/RootClusterSettingService.java
 d77b347 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java
 aef7fc1 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
 ce61497 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 8a038cc 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 3117186 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
 b8c1674 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RootStackSettingsResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java
 6a76fff 
  
ambari-server/src/main/java/org/apache/ambari/server/resources/ResourceLevelClusterSettingManager.java
 4cb6cd9 
  
ambari-server/src/main/java/org/apache/ambari/server/resources/ResourceLevelClusterSettingManagerFactory.java
 fcda654 
  
ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManager.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManagerFactory.java
 PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
878415b 
  ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
07d28ce 
  ambari-server/src/main/resources/stacks/HDP/3.0/configuration/settings.xml 
PRE-CREATION 


Diff: https://reviews.apache.org/r/63205/diff/4/

Changes: https://reviews.apache.org/r/63205/diff/3-4/


Testing
---

**Testing on 2.6 stack** (stack version in general has no effect on what we are 
testing here)


- **GET http://:/api/v1/stacks/HDP/versions/2.6/settings**


{
  "href" : "http://:/api/v1/stacks/HDP/versions/2.6/settings",
  "items" : [
{
  "href" : 
"http://:/api/v1/stacks/HDP/versions/2.6/settings/stack_features",
  "StackSettingsInfo" : {
"property_name" : "stack_features",
"stack_name" : "HDP",
"stack_version" : "2.6"
  }
},
{
  "href" : 
"http://:/api/v1/stacks/HDP/versions/2.6/settings/stack_name",
  "StackSettingsInfo" : {
"property_name" : "stack_name",
"stack_name" : "HDP",
"stack_version" : "2.6"
  }
},
{
  "href" : 
"http://:/api/v1/stacks/HDP/versions/2.6/settings/stack_packages",
  "StackSettingsInfo" : {
"property_name" : "stack_packages",
"stack_name" : "HDP",
"stack_version" : "2.6"
  }
},
{
  "href" : 
"http://:/api/v1/stacks/HDP/versions/2.6/settings/stack_root",
  "StackSettingsInfo" : {
"property_name" : "stack_root",
"stack_name" : "HDP",
"stack_version" : "2.6"
  }
},
{
  "href" : 
"http://:/api/v1/stacks/HDP/versions/2.6/settings/stack_tools",
  "StackSettingsInfo" : {
"property_name" : "stack_tools",
"stack_name" : "HDP",
"stack_version" : "2.6"
  }
}
  ]
}


- **GET

Re: Review Request 63300: HOU: Service desired repo versions are not being set

2017-10-25 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63300/#review189223
---


Ship it!





ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
Line 764 (original), 764 (patched)


Amen


- Jonathan Hurley


On Oct. 25, 2017, 12:26 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63300/
> ---
> 
> (Updated Oct. 25, 2017, 12:26 p.m.)
> 
> 
> Review request for Ambari and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-22301
> https://issues.apache.org/jira/browse/AMBARI-22301
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix the refactoring done for PU to account for HOU when setting desired 
> versions.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  8ce6454f94 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
>  aef44a5412 
> 
> 
> Diff: https://reviews.apache.org/r/63300/diff/1/
> 
> 
> Testing
> ---
> 
> Manual.  Unit tests:
> 
> [INFO] Results:
> [INFO]
> [WARNING] Tests run: 4921, Failures: 0, Errors: 0, Skipped: 34
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:05.954s
> [INFO] Finished at: Wed Oct 25 09:44:24 EDT 2017
> [INFO] Final Memory: 93M/1943M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Review Request 63300: HOU: Service desired repo versions are not being set

2017-10-25 Thread Nate Cole 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63300/
---

Review request for Ambari and Jonathan Hurley.


Bugs: AMBARI-22301
https://issues.apache.org/jira/browse/AMBARI-22301


Repository: ambari


Description
---

Fix the refactoring done for PU to account for HOU when setting desired 
versions.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 8ce6454f94 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
 aef44a5412 


Diff: https://reviews.apache.org/r/63300/diff/1/


Testing
---

Manual.  Unit tests:

[INFO] Results:
[INFO]
[WARNING] Tests run: 4921, Failures: 0, Errors: 0, Skipped: 34

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 26:05.954s
[INFO] Finished at: Wed Oct 25 09:44:24 EDT 2017
[INFO] Final Memory: 93M/1943M
[INFO] 


Thanks,

Nate Cole

Re: Review Request 63297: Alerts for OneFS mpack

2017-10-25 Thread Nate Cole 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63297/#review189202
---


Ship it!




Ship It!

- Nate Cole


On Oct. 25, 2017, 10:04 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63297/
> ---
> 
> (Updated Oct. 25, 2017, 10:04 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-22115
> https://issues.apache.org/jira/browse/AMBARI-22115
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - JXMServerSideAlert should be able to process jmx properties like: 
> java.lang:type=Memory/HeapMemoryUsage[max]
> - added 3 alerts into the mpack's alerts.json
> - fixed ams non-host metrics by adding timelineAppid=NAMENODE
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/jmx/JMXMetricHolder.java
>  f6ae54f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/alert/Reporting.java
>  a7e11e1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/jmx/JMXMetricHolderTest.java
>  61d5bae 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/alerts.json
>  5718721 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/configuration/hadoop-env.xml
>  bb671cc 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/metainfo.xml
>  f20bcf8 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/metrics.json
>  cd705eb 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/params_linux.py
>  72e0ae9 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/widgets.json
>  23da1d6 
> 
> 
> Diff: https://reviews.apache.org/r/63297/diff/1/
> 
> 
> Testing
> ---
> 
> tested manually with 3 different jmx alerts.
> 
> existing tests: passed
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 63298: Update Solr (Infra Solr + clients) version to 6.6.2 from 6.6.0

2017-10-25 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63298/#review189200
---


Ship it!




Ship It!

- Miklos Gergely


On Oct. 25, 2017, 1:58 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63298/
> ---
> 
> (Updated Oct. 25, 2017, 1:58 p.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Miklos Gergely, Robert Nettleton, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-22305
> https://issues.apache.org/jira/browse/AMBARI-22305
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Update Solr version to 6.6.2 from 6.6.0 (contains some security fixes)
> 
> 
> Diffs
> -
> 
>   ambari-infra/pom.xml 908abb4 
>   ambari-logsearch/docker/Dockerfile 1e4135e 
>   ambari-logsearch/pom.xml 9434365 
> 
> 
> Diff: https://reviews.apache.org/r/63298/diff/1/
> 
> 
> Testing
> ---
> 
> done.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 63180: Improve Kerberos principal and keytab accounting

2017-10-25 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63180/#review189196
---


Ship it!




Ship It!

- Attila Magyar


On Oct. 24, 2017, 6:02 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63180/
> ---
> 
> (Updated Oct. 24, 2017, 6:02 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Laszlo Puskas, 
> Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-22278
> https://issues.apache.org/jira/browse/AMBARI-22278
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fist step of changin way of handling keytabs.
> In this patch:
> 1. Enable cache for every principal
> 2. Changed database to represent keytab-principal relation
> 3. Created stub (ResolvedKerberosKeytab) to handle keytabs files between 
> stages in kerberos instead of principal records
> 
> Future plans:
> 1. Improve kerberos.json with support for referencing to keytab descriptor 
> and multiple principals descriptors in one identity
> 2. Refactor all *Kerberos*ServerAction.java to use KerberosKeytab instead of 
> IdentityRecord
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  2690008e59 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  29f8e2acbd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  b8e1be15d5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  4f14614000 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostKerberosIdentityResourceProvider.java
>  bfaf7b4a4f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
>  8acc401c83 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalDAO.java
>  93c55c14fa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalHostDAO.java
>  0c17f198f4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntity.java
>  bb67131584 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntityPK.java
>  600bb8b2aa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  7948a60ba2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CleanupServerAction.java
>  dae8254799 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/Component.java
>  4f1ee52739 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/ConfigureAmbariIdentitiesServerAction.java
>  fca1b6fd12 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java
>  355f515591 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java
>  1c0853b98e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosIdentityDataFile.java
>  ddf3d1b0fe 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosIdentityDataFileWriter.java
>  ea742bd940 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java
>  1b0f4fb2f9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareDisableKerberosServerAction.java
>  e1f8419b81 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareEnableKerberosServerAction.java
>  335451fa03 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareKerberosIdentitiesServerAction.java
>  038d1b5d3f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosKeytab.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/PreconfigureKerberosAction.java
>  5af7c6b35f 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceImpl.java 
> 1104d199f4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
>  3b8f6dae22 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 015ec0a9e7 
>

Re: Review Request 63265: Log Feeder: allow applying grok filter on default fields (like type, path, host etc.)

2017-10-25 Thread Robert Nettleton 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63265/#review189195
---


Ship it!




Ship It!

- Robert Nettleton


On Oct. 24, 2017, 10:30 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63265/
> ---
> 
> (Updated Oct. 24, 2017, 10:30 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert 
> Nettleton.
> 
> 
> Bugs: AMBARI-22300
> https://issues.apache.org/jira/browse/AMBARI-22300
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> add new "init_default_fields" config param to input block. If that option is 
> turned on, then logfeeder will apply  the default fields before any grok 
> filter is applied (right now it should just work with grok filter
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java
>  deff1b2 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
>  6f6bb6d 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
>  8bf6ae3 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java
>  73cf449 
> 
> 
> Diff: https://reviews.apache.org/r/63265/diff/1/
> 
> 
> Testing
> ---
> 
> UTs done
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 63298: Update Solr (Infra Solr + clients) version to 6.6.2 from 6.6.0

2017-10-25 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63298/#review189191
---


Ship it!




Ship It!

- Sebastian Toader


On Oct. 25, 2017, 3:58 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63298/
> ---
> 
> (Updated Oct. 25, 2017, 3:58 p.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Miklos Gergely, Robert Nettleton, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-22305
> https://issues.apache.org/jira/browse/AMBARI-22305
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Update Solr version to 6.6.2 from 6.6.0 (contains some security fixes)
> 
> 
> Diffs
> -
> 
>   ambari-infra/pom.xml 908abb4 
>   ambari-logsearch/docker/Dockerfile 1e4135e 
>   ambari-logsearch/pom.xml 9434365 
> 
> 
> Diff: https://reviews.apache.org/r/63298/diff/1/
> 
> 
> Testing
> ---
> 
> done.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 63298: Update Solr (Infra Solr + clients) version to 6.6.2 from 6.6.0

2017-10-25 Thread Robert Nettleton 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63298/#review189190
---


Ship it!




Ship It!

- Robert Nettleton


On Oct. 25, 2017, 1:58 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63298/
> ---
> 
> (Updated Oct. 25, 2017, 1:58 p.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Miklos Gergely, Robert Nettleton, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-22305
> https://issues.apache.org/jira/browse/AMBARI-22305
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Update Solr version to 6.6.2 from 6.6.0 (contains some security fixes)
> 
> 
> Diffs
> -
> 
>   ambari-infra/pom.xml 908abb4 
>   ambari-logsearch/docker/Dockerfile 1e4135e 
>   ambari-logsearch/pom.xml 9434365 
> 
> 
> Diff: https://reviews.apache.org/r/63298/diff/1/
> 
> 
> Testing
> ---
> 
> done.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Review Request 63297: Alerts for OneFS mpack

2017-10-25 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63297/
---

Review request for Ambari, Jonathan Hurley, Nate Cole, and Sebastian Toader.


Bugs: AMBARI-22115
https://issues.apache.org/jira/browse/AMBARI-22115


Repository: ambari


Description
---

- JXMServerSideAlert should be able to process jmx properties like: 
java.lang:type=Memory/HeapMemoryUsage[max]
- added 3 alerts into the mpack's alerts.json
- fixed ams non-host metrics by adding timelineAppid=NAMENODE


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/jmx/JMXMetricHolder.java
 f6ae54f 
  
ambari-server/src/main/java/org/apache/ambari/server/state/alert/Reporting.java 
a7e11e1 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/jmx/JMXMetricHolderTest.java
 61d5bae 
  
contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/alerts.json
 5718721 
  
contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/configuration/hadoop-env.xml
 bb671cc 
  
contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/metainfo.xml
 f20bcf8 
  
contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/metrics.json
 cd705eb 
  
contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/params_linux.py
 72e0ae9 
  
contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/widgets.json
 23da1d6 


Diff: https://reviews.apache.org/r/63297/diff/1/


Testing
---

tested manually with 3 different jmx alerts.

existing tests: passed


Thanks,

Attila Magyar

Review Request 63298: Update Solr (Infra Solr + clients) version to 6.6.2 from 6.6.0

2017-10-25 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63298/
---

Review request for Ambari, Laszlo Puskas, Miklos Gergely, Robert Nettleton, 
Sandor Magyari, and Sebastian Toader.


Bugs: AMBARI-22305
https://issues.apache.org/jira/browse/AMBARI-22305


Repository: ambari


Description
---

Update Solr version to 6.6.2 from 6.6.0 (contains some security fixes)


Diffs
-

  ambari-infra/pom.xml 908abb4 
  ambari-logsearch/docker/Dockerfile 1e4135e 
  ambari-logsearch/pom.xml 9434365 


Diff: https://reviews.apache.org/r/63298/diff/1/


Testing
---

done.


Thanks,

Oliver Szabo

Re: Review Request 63180: Improve Kerberos principal and keytab accounting

2017-10-25 Thread Eugene Chekanskiy 


> On Oct. 25, 2017, 7:28 a.m., Attila Magyar wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
> > Lines 469 (patched)
> > 
> >
> > How does regenerate missing keytabs work now? I can't find where it 
> > takes the id_distributed flag into consideration.

It works as before except that cached keytabs used always. This patch is only a 
first patch in series. Next patches will refactor how the data passed between 
stages and id_distributed will be honored when final keytabs specific to host 
are generated and distributed.


> On Oct. 25, 2017, 7:28 a.m., Attila Magyar wrote:
> > ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql
> > Lines 906 (patched)
> > 
> >
> > is there any migraction code that needs to be created for existing 
> > users?

Nope. It will added last, when refactoring and all features like many-to-many 
relation for principals-keytabs will be implemented.


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63180/#review189176
---


On Oct. 24, 2017, 6:02 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63180/
> ---
> 
> (Updated Oct. 24, 2017, 6:02 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Laszlo Puskas, 
> Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-22278
> https://issues.apache.org/jira/browse/AMBARI-22278
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fist step of changin way of handling keytabs.
> In this patch:
> 1. Enable cache for every principal
> 2. Changed database to represent keytab-principal relation
> 3. Created stub (ResolvedKerberosKeytab) to handle keytabs files between 
> stages in kerberos instead of principal records
> 
> Future plans:
> 1. Improve kerberos.json with support for referencing to keytab descriptor 
> and multiple principals descriptors in one identity
> 2. Refactor all *Kerberos*ServerAction.java to use KerberosKeytab instead of 
> IdentityRecord
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  2690008e59 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  29f8e2acbd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  b8e1be15d5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  4f14614000 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostKerberosIdentityResourceProvider.java
>  bfaf7b4a4f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
>  8acc401c83 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalDAO.java
>  93c55c14fa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalHostDAO.java
>  0c17f198f4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntity.java
>  bb67131584 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntityPK.java
>  600bb8b2aa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  7948a60ba2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CleanupServerAction.java
>  dae8254799 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/Component.java
>  4f1ee52739 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/ConfigureAmbariIdentitiesServerAction.java
>  fca1b6fd12 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java
>  355f515591 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java
>  1c0853b98e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosIdentityDataFile.java
>  ddf3d1b0fe 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosIdentityDataFileWriter.java
>  ea742bd940 
>   
>

Re: Review Request 63289: Spark history server is stopped (with umask 027 and custom spark log/pid dir)

2017-10-25 Thread Vitalyi Brodetskyi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63289/#review189177
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Жов. 25, 2017, 8:17 до полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63289/
> ---
> 
> (Updated Жов. 25, 2017, 8:17 до полудня)
> 
> 
> Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22303
> https://issues.apache.org/jira/browse/AMBARI-22303
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> 
>   1. Deploy HDP 2.4.3.0-227 on Ambari 2.5.2.0 without Spark
>   2. Enable NN HA
>   3. Upgrade Ambari to 2.6.0.0
>   4. Register install and perform RU to 2.6.3.0-220
>   5. Add Spark service
>   6. Wait for some time.  
> Result: Spark History server is stopped.
> 
> Cluster: 172.27.62.82:8080 - nat-yc-r6-pgos-ambari-hv-r-upg-1 - 48h.
> 
> Artifacts:  /ambari-hv-r-upg/split-1/nat-yc-r6-pgos-ambari-
> hv-r-upg-1/artifacts/ctr-e134-1499953498516-250582-01-14.hwx.site/artifacts/screenshots/com.hw.ambari.ui.tests.monitoring.admin_page.rolling_express_upgrade.TestRegisterAndInstallNewStackVersion/test130_AddService/_24_10_27_28_Component__Spark_History_Server__not_started_on_host_ctr_e134_1499953498516_250582_01_0/>
> 
> Spark logs:  /ambari-hv-r-upg/split-1/nat-yc-r6-pgos-ambari-hv-r-upg-1/var-
> logs/spark/ctr-e134-1499953498516-250582-01-02.hwx.site/>
> 
> From Spark.out
> 
> 
> 
> 
> The reported blocks 1677 has reached the threshold 0.9900 of total blocks 
> 1677. The number of live datanodes 5 has reached the minimum number 0. In 
> safe mode extension. Safe mode will be turned off automatically in 18 seconds.
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1422)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2693)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2582)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: 
> Cannot create file/spark-history/.342200c3-6e9e-485c-8f08-d998cd9d92aa. Name 
> node is in safe mode.
> The reported blocks 1677 has reached the threshold 0.9900 of total blocks 
> 1677. The number of live datanodes 5 has reached the minimum number 0. In 
> safe mode extension. Safe mode will be turned off automatically in 18 seconds.
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1418)
>   ... 13 more
> , while invoking ClientNamenodeProtocolTranslatorPB.create over 
> ctr-e134-1499953498516-250582-01-02.hwx.site/172.27.56.143:8020. Retrying 
> after sleeping for 22256ms.
> 
> 
> However, the NN is not in safemode state:
> 
> 
> 
> 
> [root@ctr-e134-1499953498516-250582-01-02 ~]# hdfs dfsadmin -safemode 
> get
> Safe mode is OFF in 
> ctr-e134-1499953498516-250582-01-13.hwx.site/172.27.69.83:8020
> Safe mode is OFF in 
> ctr-e134-1499953498516-250582-01-02.hwx.site/172.27.56.143:8020
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  bc86a67265 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_spark.py
>  9fc5101489 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> 922ace28b2 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 
> b4535cd02d

Review Request 63289: Spark history server is stopped (with umask 027 and custom spark log/pid dir)

2017-10-25 Thread Andrew Onischuk 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63289/
---

Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.


Bugs: AMBARI-22303
https://issues.apache.org/jira/browse/AMBARI-22303


Repository: ambari


Description
---

STR:

  1. Deploy HDP 2.4.3.0-227 on Ambari 2.5.2.0 without Spark
  2. Enable NN HA
  3. Upgrade Ambari to 2.6.0.0
  4. Register install and perform RU to 2.6.3.0-220
  5. Add Spark service
  6. Wait for some time.  
Result: Spark History server is stopped.

Cluster: 172.27.62.82:8080 - nat-yc-r6-pgos-ambari-hv-r-upg-1 - 48h.

Artifacts: 

Spark logs: 

>From Spark.out




The reported blocks 1677 has reached the threshold 0.9900 of total blocks 
1677. The number of live datanodes 5 has reached the minimum number 0. In safe 
mode extension. Safe mode will be turned off automatically in 18 seconds.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1422)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2693)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2582)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot 
create file/spark-history/.342200c3-6e9e-485c-8f08-d998cd9d92aa. Name node is 
in safe mode.
The reported blocks 1677 has reached the threshold 0.9900 of total blocks 
1677. The number of live datanodes 5 has reached the minimum number 0. In safe 
mode extension. Safe mode will be turned off automatically in 18 seconds.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1418)
... 13 more
, while invoking ClientNamenodeProtocolTranslatorPB.create over 
ctr-e134-1499953498516-250582-01-02.hwx.site/172.27.56.143:8020. Retrying 
after sleeping for 22256ms.


However, the NN is not in safemode state:




[root@ctr-e134-1499953498516-250582-01-02 ~]# hdfs dfsadmin -safemode 
get
Safe mode is OFF in 
ctr-e134-1499953498516-250582-01-13.hwx.site/172.27.69.83:8020
Safe mode is OFF in 
ctr-e134-1499953498516-250582-01-02.hwx.site/172.27.56.143:8020


Diffs
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 bc86a67265 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_spark.py
 9fc5101489 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
922ace28b2 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 
b4535cd02d 
  ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
fbe5403a35 


Diff: https://reviews.apache.org/r/63289/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 63180: Improve Kerberos principal and keytab accounting

2017-10-25 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63180/#review189176
---




ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
Lines 469 (patched)


How does regenerate missing keytabs work now? I can't find where it takes 
the id_distributed flag into consideration.



ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql
Lines 906 (patched)


is there any migraction code that needs to be created for existing users?


- Attila Magyar


On Oct. 24, 2017, 6:02 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63180/
> ---
> 
> (Updated Oct. 24, 2017, 6:02 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Laszlo Puskas, 
> Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-22278
> https://issues.apache.org/jira/browse/AMBARI-22278
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fist step of changin way of handling keytabs.
> In this patch:
> 1. Enable cache for every principal
> 2. Changed database to represent keytab-principal relation
> 3. Created stub (ResolvedKerberosKeytab) to handle keytabs files between 
> stages in kerberos instead of principal records
> 
> Future plans:
> 1. Improve kerberos.json with support for referencing to keytab descriptor 
> and multiple principals descriptors in one identity
> 2. Refactor all *Kerberos*ServerAction.java to use KerberosKeytab instead of 
> IdentityRecord
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  2690008e59 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  29f8e2acbd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  b8e1be15d5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  4f14614000 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostKerberosIdentityResourceProvider.java
>  bfaf7b4a4f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
>  8acc401c83 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalDAO.java
>  93c55c14fa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalHostDAO.java
>  0c17f198f4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntity.java
>  bb67131584 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntityPK.java
>  600bb8b2aa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  7948a60ba2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CleanupServerAction.java
>  dae8254799 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/Component.java
>  4f1ee52739 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/ConfigureAmbariIdentitiesServerAction.java
>  fca1b6fd12 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java
>  355f515591 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java
>  1c0853b98e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosIdentityDataFile.java
>  ddf3d1b0fe 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosIdentityDataFileWriter.java
>  ea742bd940 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java
>  1b0f4fb2f9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareDisableKerberosServerAction.java
>  e1f8419b81 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareEnableKerberosServerAction.java
>  335451fa03 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareKerberosIdentitiesServerAction.java
>  038d1b5d3f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosKeytab.java
>  PRE-CREATION 
>   
>

Re: Review Request 63265: Log Feeder: allow applying grok filter on default fields (like type, path, host etc.)

2017-10-25 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63265/#review189175
---


Ship it!




Ship It!

- Krisztian Kasa


On Oct. 24, 2017, 10:30 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63265/
> ---
> 
> (Updated Oct. 24, 2017, 10:30 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert 
> Nettleton.
> 
> 
> Bugs: AMBARI-22300
> https://issues.apache.org/jira/browse/AMBARI-22300
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> add new "init_default_fields" config param to input block. If that option is 
> turned on, then logfeeder will apply  the default fields before any grok 
> filter is applied (right now it should just work with grok filter
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java
>  deff1b2 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
>  6f6bb6d 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
>  8bf6ae3 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java
>  73cf449 
> 
> 
> Diff: https://reviews.apache.org/r/63265/diff/1/
> 
> 
> Testing
> ---
> 
> UTs done
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>