Re: Review Request 61197: Log Feeder properties should be handled by one class

2017-08-01 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61197/
---

(Updated Aug. 2, 2017, 4:30 a.m.)


Review request for Ambari, Oliver Szabo and Robert Nettleton.


Changes
---

Make some logical expressions in OutputSolr cleaner.


Bugs: AMBARI-21592
https://issues.apache.org/jira/browse/AMBARI-21592


Repository: ambari


Description
---

Log Feeder properties are scattered all over the code, with their 
documentation, putting large blocks of codes into several classes. Instead of 
it they should be handled by one class together with their documentation and 
default values, having all the rest of the classes asking that one for their 
value.


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
 ba3412b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
 30b61a1 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
 9535260 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
 8050263 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
 10642d1 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
 091015a 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
 7c487ba 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/loglevelfilter/LogLevelFilterHandler.java
 79bf5ea 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
 fdad9a6 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputHDFSFile.java
 2b47a00 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputS3File.java
 9f41a15 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 e3da864 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederPropertiesUtil.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java
 a244a4e 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SSLUtil.java
 573bb1c 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandlerTest.java
 c07035b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetricsManagerTest.java
 4b4067c 


Diff: https://reviews.apache.org/r/61197/diff/2/

Changes: https://reviews.apache.org/r/61197/diff/1-2/


Testing
---

Tested on docker. The properties are working fine, and the properties info REST 
API endpoint still return their description.


Thanks,

Miklos Gergely

Re: Review Request 61317: Configs For Target Stack Are Not Created During Upgrade

2017-08-01 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61317/#review181936
---




ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java
Lines 551-560 (patched)


So it turns out that in branch-2.6, we need to reintroduce this method 
since the whole idea of a cluster -> stack still exists. 

The problem with the previous implementation was that reverting by service 
config versions missed things like cluster-env which has no service.



ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
Lines 874-883 (patched)


Since the cluster still has a stack in branch-2.6, we can use 
current/desired to determine if configs need to be reverted


- Jonathan Hurley


On Aug. 1, 2017, 8:32 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61317/
> ---
> 
> (Updated Aug. 1, 2017, 8:32 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21626
> https://issues.apache.org/jira/browse/AMBARI-21626
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During an upgrade from HDP 2.5 to HDP 2.6, the new configurations are not 
> created in HDP 2.6 - they're created in HDP 2.5 which causes services to fail 
> on restart.
> 
> STR:
> - Create an HDP 2.5 cluster with ZK + HDFS
> - Do an EU to HDP 2.6
> 
> The EU will fail right after the "move stack version" step.
> 
> It looks like the bug is in {{ConfigHelper}} where it ignores the passed-in 
> {{StackId}} and uses the Cluster's stack.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
>  ff0becc822 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> b4f7120836 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 16fc971e27 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
> a5b40ff256 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  06b6217051 
> 
> 
> Diff: https://reviews.apache.org/r/61317/diff/2/
> 
> 
> Testing
> ---
> 
> Full upgrade/downgrade
> 
> UTs PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61317: Configs For Target Stack Are Not Created During Upgrade

2017-08-01 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61317/#review181928
---




ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
Lines 147-149 (patched)


Since this code was taken from trunk (where there is no stack ID of a 
cluster anymore), we needed to add this back into the branch-2.6 line...

This allows the configurations created during upgrade to be associated with 
the correct stack.



ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java
Lines 1241-1242 (patched)


Not sure what happened here with the backport, but this method was missing 
actually populating the default config types. Without this statement, we ended 
up with maps that looked like:

hdfs-site = {}, yarn-env = {}, etc

This caused empty configuration types to be created on upgrade.



ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java
Lines 893-896 (original), 893-896 (patched)


This needs to also go into trunk, as the bug exists there too - we have to 
undo the configurations for _every_ service - we should not be returning here, 
but continuing.


- Jonathan Hurley


On Aug. 1, 2017, 8:32 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61317/
> ---
> 
> (Updated Aug. 1, 2017, 8:32 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21626
> https://issues.apache.org/jira/browse/AMBARI-21626
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During an upgrade from HDP 2.5 to HDP 2.6, the new configurations are not 
> created in HDP 2.6 - they're created in HDP 2.5 which causes services to fail 
> on restart.
> 
> STR:
> - Create an HDP 2.5 cluster with ZK + HDFS
> - Do an EU to HDP 2.6
> 
> The EU will fail right after the "move stack version" step.
> 
> It looks like the bug is in {{ConfigHelper}} where it ignores the passed-in 
> {{StackId}} and uses the Cluster's stack.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
>  ff0becc822 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 16fc971e27 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
> a5b40ff256 
> 
> 
> Diff: https://reviews.apache.org/r/61317/diff/1/
> 
> 
> Testing
> ---
> 
> Full upgrade/downgrade
> 
> UTs PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Review Request 61317: Configs For Target Stack Are Not Created During Upgrade

2017-08-01 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61317/
---

Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
Lisnichenko, and Nate Cole.


Bugs: AMBARI-21626
https://issues.apache.org/jira/browse/AMBARI-21626


Repository: ambari


Description
---

During an upgrade from HDP 2.5 to HDP 2.6, the new configurations are not 
created in HDP 2.6 - they're created in HDP 2.5 which causes services to fail 
on restart.

STR:
- Create an HDP 2.5 cluster with ZK + HDFS
- Do an EU to HDP 2.6

The EU will fail right after the "move stack version" step.

It looks like the bug is in {{ConfigHelper}} where it ignores the passed-in 
{{StackId}} and uses the Cluster's stack.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
 ff0becc822 
  ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
16fc971e27 
  ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
a5b40ff256 


Diff: https://reviews.apache.org/r/61317/diff/1/


Testing
---

Full upgrade/downgrade

UTs PENDING...


Thanks,

Jonathan Hurley

Review Request 61283: AMBARI-21627. Cross-stack upgrade from IOP to HDP, ranger audit properties need to be deleted

2017-08-01 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61283/
---

Review request for Ambari, Dmytro Grinenko, Di Li, Sumit Mohanty, Sid Wagle, 
and Tim Thorpe.


Bugs: AMBARI-21627
https://issues.apache.org/jira/browse/AMBARI-21627


Repository: ambari


Description
---

STR:

* Install BigInsights 4.2.0 with Ranger, Knox, Kafka, Hive, HBase, YARN, HDFS
* Upgrade Ambari to 2.5.2
* Install HDP 2.6
* Perform EU to HDP 2.6


All of the config types like ranger-*-audit.xml have configs that need to be 
deleted that begin with xasecure.audit.destination.db.*
E.g.,
```
xasecure.audit.destination.db: "true",
xasecure.audit.destination.db.batch.filespool.dir: 
"/var/log/hbase/audit/db/spool",
xasecure.audit.destination.db.jdbc.driver: "{{jdbc_driver}}",
xasecure.audit.destination.db.jdbc.url: "{{audit_jdbc_url}}",
xasecure.audit.destination.db.password: 
"SECRET:ranger-hbase-audit:3:xasecure.audit.destination.db.password",
xasecure.audit.destination.db.user: "{{xa_audit_db_user}}",
```


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 17e80b5 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 55736cf 


Diff: https://reviews.apache.org/r/61283/diff/1/


Testing
---

Inspected manually


Thanks,

Alejandro Fernandez

Re: Review Request 61209: Log Search should stop using deprecated API for solr

2017-08-01 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61209/#review181908
---


Ship it!




Ship It!

- Alejandro Fernandez


On July 28, 2017, 12:25 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61209/
> ---
> 
> (Updated July 28, 2017, 12:25 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
> 
> 
> Bugs: AMBARI-21596
> https://issues.apache.org/jira/browse/AMBARI-21596
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Since we've switched to solrj 6.6.0 some API calls we are using are 
> deprecated now. Let's use non-deprecated API instead.
> 
> Also suppressed some warnings and fixed the formatting of the pom file.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-server/pom.xml ebca2d5 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/SolrAuditAliasConfigurer.java
>  1eca94b 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/SolrCollectionConfigurer.java
>  225f5a3 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/SolrSchemaFieldDao.java
>  71f9f29 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/AbstractSolrConfigHandler.java
>  6f3d8ca 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/CreateCollectionHandler.java
>  b6e9def 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/ReloadCollectionHandler.java
>  52f3366 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/ResponseDataGenerator.java
>  74e9c11 
> 
> 
> Diff: https://reviews.apache.org/r/61209/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on docker, still works :)
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 61201: AMBARI-21594. MultiEverything : Add Servicegroup as a subresource of Cluster.

2017-08-01 Thread Jayush Luniya 


> On Aug. 1, 2017, 5:51 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceGroupResourceProvider.java
> > Lines 234 (patched)
> > 
> >
> > Shouldn't this be
> > for(ServiceGroupRequest request : requests) {
> >   deleteStatusMetaData.addDeletedKey(request.getClusterName() + "/" + 
> > request.getServiceGroupName());
> > }
> 
> Swapan Shridhar wrote:
> request already has these 2 values only. 
> This is the result with : 'requests.toString()'
> 
> 
> {
>   "deleteResult" : [
> {
>   "deleted" : {
> "key" : "[clusterName=c1, serviceGroupName=CORE]"
>   }
> }
>   ]
> }

what I meant was for bulk delete should it be adding a deleted key for each?


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61201/#review181877
---


On July 31, 2017, 9:06 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61201/
> ---
> 
> (Updated July 31, 2017, 9:06 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.
> 
> 
> Bugs: AMBARI-21594
> https://issues.apache.org/jira/browse/AMBARI-21594
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> **ServiceGroup (SG)** : is defined as sub-resource of cluster. It would later 
> be used to logically group a set of services (coming out of Management Pack 
> selected).
>  
> **Example of a Service groups can be a:** 
> - “Core” Service Group  containing HDFS and Zookeeper. This will be formed by 
> default.
> - “Streaming” Service Group containing Kafka, Storm, Nifi.
> - “Data Science” Service Group containing Hive, Spark, Zeppelin.
> - “EDW” Service Group containing Hive, Spark.
> 
> As part of this review, following has been implemented:
> - SG added as sub-resource of cluster.
> - GET, POST and DELETE API for SG.
> 
> 
> 
> *API calls and response:*
> 
> *1.* 
> 
> POST:
> 
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups
> *Body :*
> 
> 
> [
> {
>   "ServiceGroupInfo" : {
> "cluster_id": "2",
> "service_group_name": "CORE"
>   }
> },
> {
>   "ServiceGroupInfo" : {
> "cluster_id": "2",
> "service_group_name": "EDW"
>   }
> }
> ]
> 
> 
> *Response:* 201 CREATED
> 
> 
> {
>   "resources" : [
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE";,
>   "ServiceGroupInfo" : {
> "cluster_id" : 2,
> "cluster_name" : "c1",
> "service_group_id" : 804,
> "service_group_name" : "CORE"
>   }
> },
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/EDW";,
>   "ServiceGroupInfo" : {
> "cluster_id" : 2,
> "cluster_name" : "c1",
> "service_group_id" : 803,
> "service_group_name" : "EDW"
>   }
> }
>   ]
> }
> 
> 
> *2.*
> ===
> GET :
> ===
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/
> 
> *Body:*
> 
> []
> 
> 
> *Response:* 200 OK
> 
> 
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/";,
>   "items" : [
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE";,
>   "ServiceGroupInfo" : {
> "cluster_name" : "c1",
> "service_group_name" : "CORE"
>   }
> },
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/EDW";,
>   "ServiceGroupInfo" : {
> "cluster_name" : "c1",
> "service_group_name" : "EDW"
>   }
> }
>   ]
> }
> 
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE
> 
> *Body:*
> 
> []
> 
> 
> *Response:* 200 OK
> 
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE";,
>   "ServiceGroupInfo" : {
> "cluster_id" : 2,
> "cluster_name" : "c1",
> "service_group_id" : 804,
> "service_group_name" : "CORE"
>   }
> }
> 
> 
> *3.*
> ===
> DELETE:
> ===
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE
> 
> *Body:*
> 
> []
> 
> 
> *Response:* 200 OK
> 
> 
> {
>   "deleteResult" : [
> {
>   "deleted" : {
> "key" : "[clusterName=c1, serviceGroupName=CORE]"
>   }
> }
>   ]
> }
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari

Re: Review Request 61276: Replace Hard Coded conf-select Structures

2017-08-01 Thread Jonathan Hurley 


> On Aug. 1, 2017, 1:12 p.m., Dmitro Lisnichenko wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
> > Line 24 (original), 24 (patched)
> > 
> >
> > I think we usually do
> > 
> > import ambari_simplejson as json # simplejson is much faster comparing 
> > to Python 2.6 json module and has the same functions set.

Ah, yes, good catch. I'll make the change.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61276/#review181903
---


On Aug. 1, 2017, 10:09 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61276/
> ---
> 
> (Updated Aug. 1, 2017, 10:09 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21581
> https://issues.apache.org/jira/browse/AMBARI-21581
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, both the stack-select and conf-select python files have hard-coded 
> structures inside of them which contain mappings for things like 
> configuration directories and stack-select component names. 
> 
> This code is really HDP-specific in that it reverses the mappings of the RPMs 
> and allows us to remap {{/etc//conf}} to 
> {{/usr/hdp/current//conf}}
> {code:title=conf_select.py}
> _PACKAGE_DIRS = {
>   "atlas": [
> {
>   "conf_dir": "/etc/atlas/conf",
>   "current_dir": 
> "{0}/current/atlas-client/conf".format(STACK_ROOT_PATTERN)
> }
>   ],
>   "accumulo": [
> {
>   "conf_dir": "/etc/accumulo/conf",
>   "current_dir": 
> "{0}/current/accumulo-client/conf".format(STACK_ROOT_PATTERN)
> }
>   ],
> ...
> {code}
> 
> With the coming of management packs replacing stacks, we can no longer hard 
> code this in Python.
> 
> Additionally, there is an odd anti-pattern here where we conf-select on 
> installation and also on upgrade. The upgrade should never deal with 
> conf-select as those directories should have already been created during the 
> installation phase.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  4f11633fbe 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  9e28907f8f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  ec8b38c9a7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 827037056d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java
>  55f9665e32 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  856446ccce 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
>  a0519b20da 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  6e4a168881 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py
>  ff7bf5fe88 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/druid_node.py
>  20623f7f5e 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/superset.py
>  36dab51441 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  85d80ba6a5 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py
>  c7ba556304 
>   
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py
>  f57fae7b37 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
>  4820d24883 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/phoenix_queryserver.py
>  f2b44bddeb 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/upgrade.py
>  8ea3616ecc 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py
>  ba926cb56e 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
>  f320eee075 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py
>  402e7ade26 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  aac7977b6b 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package

Re: Review Request 61201: AMBARI-21594. MultiEverything : Add Servicegroup as a subresource of Cluster.

2017-08-01 Thread Swapan Shridhar 


> On Aug. 1, 2017, 5:51 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceGroupResourceProvider.java
> > Lines 234 (patched)
> > 
> >
> > Shouldn't this be
> > for(ServiceGroupRequest request : requests) {
> >   deleteStatusMetaData.addDeletedKey(request.getClusterName() + "/" + 
> > request.getServiceGroupName());
> > }

request already has these 2 values only. 
This is the result with : 'requests.toString()'


{
  "deleteResult" : [
{
  "deleted" : {
"key" : "[clusterName=c1, serviceGroupName=CORE]"
  }
}
  ]
}


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61201/#review181877
---


On July 31, 2017, 9:06 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61201/
> ---
> 
> (Updated July 31, 2017, 9:06 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.
> 
> 
> Bugs: AMBARI-21594
> https://issues.apache.org/jira/browse/AMBARI-21594
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> **ServiceGroup (SG)** : is defined as sub-resource of cluster. It would later 
> be used to logically group a set of services (coming out of Management Pack 
> selected).
>  
> **Example of a Service groups can be a:** 
> - “Core” Service Group  containing HDFS and Zookeeper. This will be formed by 
> default.
> - “Streaming” Service Group containing Kafka, Storm, Nifi.
> - “Data Science” Service Group containing Hive, Spark, Zeppelin.
> - “EDW” Service Group containing Hive, Spark.
> 
> As part of this review, following has been implemented:
> - SG added as sub-resource of cluster.
> - GET, POST and DELETE API for SG.
> 
> 
> 
> *API calls and response:*
> 
> *1.* 
> 
> POST:
> 
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups
> *Body :*
> 
> 
> [
> {
>   "ServiceGroupInfo" : {
> "cluster_id": "2",
> "service_group_name": "CORE"
>   }
> },
> {
>   "ServiceGroupInfo" : {
> "cluster_id": "2",
> "service_group_name": "EDW"
>   }
> }
> ]
> 
> 
> *Response:* 201 CREATED
> 
> 
> {
>   "resources" : [
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE";,
>   "ServiceGroupInfo" : {
> "cluster_id" : 2,
> "cluster_name" : "c1",
> "service_group_id" : 804,
> "service_group_name" : "CORE"
>   }
> },
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/EDW";,
>   "ServiceGroupInfo" : {
> "cluster_id" : 2,
> "cluster_name" : "c1",
> "service_group_id" : 803,
> "service_group_name" : "EDW"
>   }
> }
>   ]
> }
> 
> 
> *2.*
> ===
> GET :
> ===
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/
> 
> *Body:*
> 
> []
> 
> 
> *Response:* 200 OK
> 
> 
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/";,
>   "items" : [
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE";,
>   "ServiceGroupInfo" : {
> "cluster_name" : "c1",
> "service_group_name" : "CORE"
>   }
> },
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/EDW";,
>   "ServiceGroupInfo" : {
> "cluster_name" : "c1",
> "service_group_name" : "EDW"
>   }
> }
>   ]
> }
> 
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE
> 
> *Body:*
> 
> []
> 
> 
> *Response:* 200 OK
> 
> {
>   "href" : 
> "http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE";,
>   "ServiceGroupInfo" : {
> "cluster_id" : 2,
> "cluster_name" : "c1",
> "service_group_id" : 804,
> "service_group_name" : "CORE"
>   }
> }
> 
> 
> *3.*
> ===
> DELETE:
> ===
> 
> API: http://c6404.ambari.apache.org:8080/api/v1/clusters/c1/servicegroups/CORE
> 
> *Body:*
> 
> []
> 
> 
> *Response:* 200 OK
> 
> 
> {
>   "deleteResult" : [
> {
>   "deleted" : {
> "key" : "[clusterName=c1, serviceGroupName=CORE]"
>   }
> }
>   ]
> }
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/ServiceGroupNotFoundException.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ClusterResourceDefinition.java
>  f689841 
>   
>

Re: Review Request 61276: Replace Hard Coded conf-select Structures

2017-08-01 Thread Dmitro Lisnichenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61276/#review181903
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
Line 24 (original), 24 (patched)


I think we usually do

import ambari_simplejson as json # simplejson is much faster comparing to 
Python 2.6 json module and has the same functions set.


- Dmitro Lisnichenko


On Aug. 1, 2017, 5:09 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61276/
> ---
> 
> (Updated Aug. 1, 2017, 5:09 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
> Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-21581
> https://issues.apache.org/jira/browse/AMBARI-21581
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, both the stack-select and conf-select python files have hard-coded 
> structures inside of them which contain mappings for things like 
> configuration directories and stack-select component names. 
> 
> This code is really HDP-specific in that it reverses the mappings of the RPMs 
> and allows us to remap {{/etc//conf}} to 
> {{/usr/hdp/current//conf}}
> {code:title=conf_select.py}
> _PACKAGE_DIRS = {
>   "atlas": [
> {
>   "conf_dir": "/etc/atlas/conf",
>   "current_dir": 
> "{0}/current/atlas-client/conf".format(STACK_ROOT_PATTERN)
> }
>   ],
>   "accumulo": [
> {
>   "conf_dir": "/etc/accumulo/conf",
>   "current_dir": 
> "{0}/current/accumulo-client/conf".format(STACK_ROOT_PATTERN)
> }
>   ],
> ...
> {code}
> 
> With the coming of management packs replacing stacks, we can no longer hard 
> code this in Python.
> 
> Additionally, there is an odd anti-pattern here where we conf-select on 
> installation and also on upgrade. The upgrade should never deal with 
> conf-select as those directories should have already been created during the 
> installation phase.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  4f11633fbe 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  9e28907f8f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  ec8b38c9a7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
> 827037056d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java
>  55f9665e32 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  856446ccce 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
>  a0519b20da 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  6e4a168881 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py
>  ff7bf5fe88 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/druid_node.py
>  20623f7f5e 
>   
> ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/superset.py
>  36dab51441 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  85d80ba6a5 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py
>  c7ba556304 
>   
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py
>  f57fae7b37 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
>  4820d24883 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/phoenix_queryserver.py
>  f2b44bddeb 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/upgrade.py
>  8ea3616ecc 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py
>  ba926cb56e 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
>  f320eee075 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py
>  402e7ade26 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  aac7977b6b 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/nfsgateway.py
>  dea9d43d16 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scri

Re: Review Request 61209: Log Search should stop using deprecated API for solr

2017-08-01 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61209/#review181902
---


Ship it!




Ship It!

- Oliver Szabo


On July 28, 2017, 12:25 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61209/
> ---
> 
> (Updated July 28, 2017, 12:25 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
> 
> 
> Bugs: AMBARI-21596
> https://issues.apache.org/jira/browse/AMBARI-21596
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Since we've switched to solrj 6.6.0 some API calls we are using are 
> deprecated now. Let's use non-deprecated API instead.
> 
> Also suppressed some warnings and fixed the formatting of the pom file.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-server/pom.xml ebca2d5 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/SolrAuditAliasConfigurer.java
>  1eca94b 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/SolrCollectionConfigurer.java
>  225f5a3 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/SolrSchemaFieldDao.java
>  71f9f29 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/AbstractSolrConfigHandler.java
>  6f3d8ca 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/CreateCollectionHandler.java
>  b6e9def 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/ReloadCollectionHandler.java
>  52f3366 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/ResponseDataGenerator.java
>  74e9c11 
> 
> 
> Diff: https://reviews.apache.org/r/61209/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on docker, still works :)
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review181897
---


Ship it!




Ship It!

- Sebastian Toader


On Aug. 1, 2017, 4:56 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 1, 2017, 4:56 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/2/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 61277: Log Search Config should be separated into a Server and Log Feeder interface

2017-08-01 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61277/#review181896
---




ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigLogFeederZK.java
Lines 99 (patched)


its not an issue and we wont solve that here, im just mentioning it, so in 
the future maybe we should use config_group or something like that instead of 
cluster name (in terms of an ambari cluster that will be the same, but it will 
means something else in the config api)


- Oliver Szabo


On Aug. 1, 2017, 3:09 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61277/
> ---
> 
> (Updated Aug. 1, 2017, 3:09 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
> 
> 
> Bugs: AMBARI-21623
> https://issues.apache.org/jira/browse/AMBARI-21623
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Log Search Config is getting too complex, it serves too many kind of things. 
> It should be separated into two interfaces, one for the Server and one for 
> the Log Feeders to make it cleaner.
> 
> Also fixed a minor issue about shipper configurations without any post map 
> values.
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
>  76be392 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  77b48eb 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigServer.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass1.java
>  e308346 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass2.java
>  b64dae8 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactoryTest.java
>  f990c5c 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeederClass1.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeederClass2.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigServerClass1.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigServerClass2.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigLogFeederZK.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigServerZK.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigZK.java
>  387d0c6 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  ba3412b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  30b61a1 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
>  10642d1 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/Output.java
>  b370e58 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
>  ce040f9 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/LogSearchConfigConfigurer.java
>  3f6df75 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/ShipperConfigManager.java
>  a1181b4 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerFilter.java
>  84bddb8 
> 
> 
> Diff: https://reviews.apache.org/r/61277/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on local cluster, still works
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Review Request 61277: Log Search Config should be separated into a Server and Log Feeder interface

2017-08-01 Thread Miklos Gergely 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61277/
---

Review request for Ambari, Oliver Szabo and Robert Nettleton.


Bugs: AMBARI-21623
https://issues.apache.org/jira/browse/AMBARI-21623


Repository: ambari


Description
---

Log Search Config is getting too complex, it serves too many kind of things. It 
should be separated into two interfaces, one for the Server and one for the Log 
Feeders to make it cleaner.

Also fixed a minor issue about shipper configurations without any post map 
values.


Diffs
-

  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
 76be392 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
 77b48eb 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigServer.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass1.java
 e308346 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass2.java
 b64dae8 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactoryTest.java
 f990c5c 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeederClass1.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeederClass2.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigServerClass1.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigServerClass2.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigLogFeederZK.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigServerZK.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigZK.java
 387d0c6 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
 ba3412b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
 30b61a1 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
 10642d1 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/Output.java
 b370e58 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
 ce040f9 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/LogSearchConfigConfigurer.java
 3f6df75 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/ShipperConfigManager.java
 a1181b4 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/common/LSServerFilter.java
 84bddb8 


Diff: https://reviews.apache.org/r/61277/diff/1/


Testing
---

Tested on local cluster, still works


Thanks,

Miklos Gergely

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/
---

(Updated Aug. 1, 2017, 2:56 p.m.)


Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
Toader.


Changes
---

- os.path.isfile ->sudo.path_exists
- MISSING_KEYTABS -> CHECK_KEYTABS
- added javadoc, fixd typo


Bugs: AMBARI-21613
https://issues.apache.org/jira/browse/AMBARI-21613


Repository: ambari


Description
---

Dynamically determine what keytab files have been distributed to hosts. A 
custom command should be available via the KERBEROS_CLIENT to query for the 
keytab files installed on the relevant host. The communication between the 
Ambari server and the agents should generate data needed to determine what 
keytab files exist.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 1bc4c36 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
 a08abab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 e5b7afd 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 6a2dd09 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
 39fdcf5 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
 fcd57af 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml 
0e42bda 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
 b2cdaa6 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
 abf58ee 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
 baa9bae 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 4508527 
  ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
f638845 


Diff: https://reviews.apache.org/r/61251/diff/2/

Changes: https://reviews.apache.org/r/61251/diff/1-2/


Testing
---

1.
- created a cluster with kerberos enabled
- deleted a keytab file from a host
- regenerated missing keytabs
- checked that the deleted keytab file was recreated

2.
- regenerated missing keytabs when no keytab files were missing
- checked that no keytab file was regenerated

3.
- created a 15 nodes cluster on openstack with services: HDFS, YARN, ZOOKEEPER, 
SPARK1, SPARK2, HIVE
- measured the time taken to regenerate all keytabs and only missing keytabs:

  regenerate missing keytabs only: 12.84 seconds (?=2.33)
  regenerate all keytabs: 39.1 seconds (?=5.37)

existing tests: pending


Thanks,

Attila Magyar

Review Request 61276: Replace Hard Coded conf-select Structures

2017-08-01 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61276/
---

Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro 
Lisnichenko, and Nate Cole.


Bugs: AMBARI-21581
https://issues.apache.org/jira/browse/AMBARI-21581


Repository: ambari


Description
---

Currently, both the stack-select and conf-select python files have hard-coded 
structures inside of them which contain mappings for things like configuration 
directories and stack-select component names. 

This code is really HDP-specific in that it reverses the mappings of the RPMs 
and allows us to remap {{/etc//conf}} to 
{{/usr/hdp/current//conf}}
{code:title=conf_select.py}
_PACKAGE_DIRS = {
  "atlas": [
{
  "conf_dir": "/etc/atlas/conf",
  "current_dir": "{0}/current/atlas-client/conf".format(STACK_ROOT_PATTERN)
}
  ],
  "accumulo": [
{
  "conf_dir": "/etc/accumulo/conf",
  "current_dir": 
"{0}/current/accumulo-client/conf".format(STACK_ROOT_PATTERN)
}
  ],
...
{code}

With the coming of management packs replacing stacks, we can no longer hard 
code this in Python.

Additionally, there is an odd anti-pattern here where we conf-select on 
installation and also on upgrade. The upgrade should never deal with 
conf-select as those directories should have already been created during the 
installation phase.


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 4f11633fbe 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 9e28907f8f 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 ec8b38c9a7 
  ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
827037056d 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java
 55f9665e32 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
 856446ccce 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
 a0519b20da 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
 6e4a168881 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py
 ff7bf5fe88 
  
ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/druid_node.py
 20623f7f5e 
  
ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/superset.py
 36dab51441 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
 85d80ba6a5 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py
 c7ba556304 
  
ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py
 f57fae7b37 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
 4820d24883 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/phoenix_queryserver.py
 f2b44bddeb 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/upgrade.py
 8ea3616ecc 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py
 ba926cb56e 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
 f320eee075 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py
 402e7ade26 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
 aac7977b6b 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/nfsgateway.py
 dea9d43d16 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/snamenode.py
 115a830966 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py
 fabbabbac6 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py
 1694816594 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py
 c84f47e074 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py
 a1212e00c9 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py
 16d7907c57 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py
 db86d9ec5e 
  
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py
 08c3dd19de 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py
 6df68ccd3e 
  
ambari-server/src/main/re

Re: Review Request 61269: AMBARI-21619. More ResourceManager HA host group placeholders in blueprints

2017-08-01 Thread Robert Nettleton 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61269/#review181893
---


Ship it!




Ship It!

- Robert Nettleton


On Aug. 1, 2017, 7:04 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61269/
> ---
> 
> (Updated Aug. 1, 2017, 7:04 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21619
> https://issues.apache.org/jira/browse/AMBARI-21619
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Replace placeholders for few more properties.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  b0a605be45c540566240bbb032fd220d3d051859 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  c5af0d9962f076d5681b9c9550aef210f75efb10 
> 
> 
> Diff: https://reviews.apache.org/r/61269/diff/1/
> 
> 
> Testing
> ---
> 
> Tested blueprint install of HA cluster.
> Unit tests: changed related test; tests pass.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review181891
---



This looks great. I like the concept.


ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
Lines 145 (patched)


How about renaming this to something like `CHECK_KEYTAB`.  It is more like 
an actions like... SET and REMOVE.



ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
Lines 3047 (patched)


`addGettingMisingKeytabsStage` --> `addGettingMissingKeytabsStage`  
(spelling issue: Mising)

Also, missing JavaDoc.



ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
Lines 456-457 (patched)


In the (not so distanct) future we may want to place multiple principals in 
the same keytab file, or at least allow the user to set the configurations for 
this.

It appears that the input and output structures should support this, 
however the test is not checking for principal. It only checks for the 
existance of the keytab file.  It would be nice to make sure the specifiec 
principal exists in the specified keytab file. Additional work will be needed 
if we want to verify that the requied keytabs are in the keytab file.  For 
example AES256, RC4, etc... 

However, for now, maybe a `klist` needs to be performed on the keytab file 
to ensure at least one keytab entry for the requested principal exists. 

Sonehow this should be done via sudo in the event the agent is not running 
as root.  Maybe a KeytabFile resource can be created for this? See 
`resource_management.core.providers.system.FileProvider` and 
`resource_management.core.resources.system.File`.


- Robert Levas


On Aug. 1, 2017, 5:58 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 1, 2017, 5:58 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/1/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila M

Re: Review Request 61269: AMBARI-21619. More ResourceManager HA host group placeholders in blueprints

2017-08-01 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61269/#review181889
---


Ship it!




Ship It!

- Jonathan Hurley


On Aug. 1, 2017, 3:04 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61269/
> ---
> 
> (Updated Aug. 1, 2017, 3:04 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21619
> https://issues.apache.org/jira/browse/AMBARI-21619
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Replace placeholders for few more properties.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  b0a605be45c540566240bbb032fd220d3d051859 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  c5af0d9962f076d5681b9c9550aef210f75efb10 
> 
> 
> Diff: https://reviews.apache.org/r/61269/diff/1/
> 
> 
> Testing
> ---
> 
> Tested blueprint install of HA cluster.
> Unit tests: changed related test; tests pass.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review181888
---




ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
Lines 471 (patched)


If ambari-agent runs as non-root user, will the agent have access to the 
path of the key tab file to verify it exists? Note access to a keytab file may 
be restricted to specific users and groups (defined in kerberos.json).


- Sebastian Toader


On Aug. 1, 2017, 11:58 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 1, 2017, 11:58 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/1/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 61269: AMBARI-21619. More ResourceManager HA host group placeholders in blueprints

2017-08-01 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61269/#review181887
---


Ship it!




Ship It!

- Sebastian Toader


On Aug. 1, 2017, 9:04 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61269/
> ---
> 
> (Updated Aug. 1, 2017, 9:04 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21619
> https://issues.apache.org/jira/browse/AMBARI-21619
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Replace placeholders for few more properties.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  b0a605be45c540566240bbb032fd220d3d051859 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  c5af0d9962f076d5681b9c9550aef210f75efb10 
> 
> 
> Diff: https://reviews.apache.org/r/61269/diff/1/
> 
> 
> Testing
> ---
> 
> Tested blueprint install of HA cluster.
> Unit tests: changed related test; tests pass.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/
---

Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
Toader.


Bugs: AMBARI-21613
https://issues.apache.org/jira/browse/AMBARI-21613


Repository: ambari


Description
---

Dynamically determine what keytab files have been distributed to hosts. A 
custom command should be available via the KERBEROS_CLIENT to query for the 
keytab files installed on the relevant host. The communication between the 
Ambari server and the agents should generate data needed to determine what 
keytab files exist.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 1bc4c36 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
 a08abab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 e5b7afd 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 6a2dd09 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
 39fdcf5 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
 fcd57af 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml 
0e42bda 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
 b2cdaa6 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
 abf58ee 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
 baa9bae 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 4508527 
  ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
f638845 


Diff: https://reviews.apache.org/r/61251/diff/1/


Testing
---

1.
- created a cluster with kerberos enabled
- deleted a keytab file from a host
- regenerated missing keytabs
- checked that the deleted keytab file was recreated

2.
- regenerated missing keytabs when no keytab files were missing
- checked that no keytab file was regenerated

3.
- created a 15 nodes cluster on openstack with services: HDFS, YARN, ZOOKEEPER, 
SPARK1, SPARK2, HIVE
- measured the time taken to regenerate all keytabs and only missing keytabs:

  regenerate missing keytabs only: 12.84 seconds (?=2.33)
  regenerate all keytabs: 39.1 seconds (?=5.37)

existing tests: pending


Thanks,

Attila Magyar

Review Request 61273: Do not call refreshnodes command after deleting a slave

2017-08-01 Thread Dmytro Sen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61273/
---

Review request for Ambari, Sumit Mohanty, Sid Wagle, and Vitalyi Brodetskyi.


Bugs: AMBARI-21621
https://issues.apache.org/jira/browse/AMBARI-21621


Repository: ambari


Description
---

If DN is deleted the hostname is removed from exclude file and refreshnodes 
command is called. If those actions are not followed by NN restart, the deleted 
DN will be shown as DEAD on NN UI.

The slave should be removed by NN from the UI, but Ambari should not call 
refreshnodes command on deletion to ensure slave is in "decommissioned and 
dead" state, not "dead"


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 36cfb0f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 2b8bf93 


Diff: https://reviews.apache.org/r/61273/diff/1/


Testing
---

manual + unit tests


Thanks,

Dmytro Sen

Review Request 61269: AMBARI-21619. More ResourceManager HA host group placeholders in blueprints

2017-08-01 Thread Attila Doroszlai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61269/
---

Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sebastian 
Toader.


Bugs: AMBARI-21619
https://issues.apache.org/jira/browse/AMBARI-21619


Repository: ambari


Description
---

Replace placeholders for few more properties.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 b0a605be45c540566240bbb032fd220d3d051859 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
 c5af0d9962f076d5681b9c9550aef210f75efb10 


Diff: https://reviews.apache.org/r/61269/diff/1/


Testing
---

Tested blueprint install of HA cluster.
Unit tests: changed related test; tests pass.


Thanks,

Attila Doroszlai