Re: Review Request 58109: Large Long Running Requests Can Slow Down the ActionScheduler

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58109/
---

(Updated March 31, 2017, 9:16 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Bugs: AMBARI-20646
https://issues.apache.org/jira/browse/AMBARI-20646


Repository: ambari


Description
---

When creating a massive request (a rolling upgrade on a cluster with 1000 
nodes), the size of the request seems to slow down the {{ActionScheduler}}. 
Each command was taking between 1 to 2 minutes to run (even server-side tasks). 

The cause of this can be seen in the following two stack traces:

{code:title=ActionSchedulerImpl}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:303)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:341)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
at java.lang.Thread.run(Thread.java:745)
{code}

{code:title=Server Action Executor}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:700)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.Request.(Request.java:199)
at 
org.apache.ambari.server.actionmanager.Request$$FastClassByGuice$$9071e03.newInstance()
at 
com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at 
com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
at 
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at 
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at 
com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
at 
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
at 
com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
at com.sun.proxy.$Proxy26.createExisting(Unknown Source)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getRequests(ActionDBAccessorImpl.java:784)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.cleanRequestShareDataContexts(ServerActionExecutor.java:259)
- locked <0x7ff0a14083c8> (a java.util.HashMap)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.doWork(ServerActionExecutor.java:454)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor$1.run(ServerActionExecutor.java:160)
at java.lang.Thread.run(Thread.java:745)
{code}

It's clear from these stacks that every {{PENDING}} stage (roughly 15,000) were 
being loaded into memory every second (and their accompanying task as well). 
This makes no sense as these methods don't need all stages - just the _next_ 
stage. This is because all stages are synchronous within a single request.

The proposed solution is to fix the {{StageEntity.findByCommandStatuses}} call 
so it doesn't return every stage:
{code}
SELECT stage.requestid, 
   MIN(stage.stageid) 
FROM   stageentity stage, 
   hostrolecommandentity hrc 
WHERE  hrc.status IN :statuses 
   AND hrc.stageid = stage.stageid 
   AND hrc.requestid = stage.requestid 
GROUP  BY stage.requestid 
{code}


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessor.java
 9325d03 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
 ab4feaa 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java
 0984c5c 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java 
5151fb3 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java
 f68338f 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java
 b0be6b3 
  
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java
 81eef3b 
  

Review Request 58129: HDP 3.0 TP - add Pig client: configs, metainfo, descriptors, service advisor and etc

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58129/
---

Review request for Ambari and Alejandro Fernandez.


Bugs: AMBARI-20652
https://issues.apache.org/jira/browse/AMBARI-20652


Repository: ambari


Description
---

Add Pig client to HDP 3.0


Diffs
-

  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/configuration/pig-env.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/configuration/pig-log4j.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/configuration/pig-properties.xml
 PRE-CREATION 
  ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/kerberos.json 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/metainfo.xml 
PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/package/files/pigSmoke.sh
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/package/scripts/params.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/package/scripts/params_linux.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/package/scripts/params_windows.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/package/scripts/pig.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/package/scripts/pig_client.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/package/scripts/service_check.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/PIG/0.16.1.3.0/service_advisor.py
 PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/3.0/services/PIG/metainfo.xml 
PRE-CREATION 


Diff: https://reviews.apache.org/r/58129/diff/1/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Review Request 58124: AMBARI-20651 - When in Upgrade paused state, links should be disabled for less privileged user

[Richard Zang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58124/
---

Review request for Ambari, Jaimin Jetly and Xi Wang.


Bugs: AMBARI-20651
https://issues.apache.org/jira/browse/AMBARI-20651


Repository: ambari


Description
---

Show not allowed cursor for user without privilege to access upgrade wizard.


Diffs
-

  ambari-web/app/controllers/application.js df0a8e8 


Diff: https://reviews.apache.org/r/58124/diff/1/


Testing
---

Manually tested on live cluster. All unit tests passed.
  20690 passing (22s)
  128 pending


Thanks,

Richard Zang

Review Request 58122: Blueprint export fails if config-type is not mapped to any service after upgrade

[Amruta Borkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58122/
---

Review request for Ambari, Di Li and Robert Nettleton.


Bugs: AMBARI-20551
https://issues.apache.org/jira/browse/AMBARI-20551


Repository: ambari


Description
---

Following error is thrown while exporting a blueprint from cluster.
 ERROR [ambari-client-thread-8726] ReadHandler:99 - Bad request:
java.lang.IllegalArgumentException: Specified configuration type is not 
associated with any service: storm-site
at 
org.apache.ambari.server.controller.internal.Stack.getServiceForConfigType(Stack.java:494)
at 
org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor$StackPropertyTypeFilter.isPropertyIncluded(BlueprintConfigurationProcessor.java:2946)
at 
org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.shouldPropertyBeExcludedForBlueprintExport(BlueprintConfigurationProcessor.java:939)
at 
org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doFilterPriorToExport(BlueprintConfigurationProcessor.java:439)
at 
org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doUpdateForBlueprintExport(BlueprintConfigurationProcessor.java:416)
at 
org.apache.ambari.server.api.query.render.ClusterBlueprintRenderer.createBlueprintResource(ClusterBlueprintRenderer.java:186)
at 
org.apache.ambari.server.api.query.render.ClusterBlueprintRenderer.finalizeResult(ClusterBlueprintRenderer.java:141)
at org.apache.ambari.server.api.query.QueryImpl.getResult(QueryImpl.java:839)
at org.apache.ambari.server.api.query.QueryImpl.execute(QueryImpl.java:223)
at 
org.apache.ambari.server.api.handlers.ReadHandler.handleRequest(ReadHandler.java:77)
at 
org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:145)


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 25f2001 


Diff: https://reviews.apache.org/r/58122/diff/1/


Testing
---

Tested manually. Log shows warning about unmapped config-type but blueprint 
gets exported.


Thanks,

Amruta Borkar

Re: Review Request 58106: AMBARI-20636 Allow users to add custom configs for Ranger service in all plugins

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58106/#review170795
---


Ship it!




Ship It!

- Alejandro Fernandez


On March 31, 2017, 12:21 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58106/
> ---
> 
> (Updated March 31, 2017, 12:21 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-20636
> https://issues.apache.org/jira/browse/AMBARI-20636
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In order to help Ranger to communicate with Hive metastore (in cloud env) 
> there is a need to allow users to add custom configs from Ambari, which needs 
> to be added as part of Ranger service which gets created on Enabling plugins.
> This will be helpful for all plugins to define n-number of custom configs 
> from Ambari.
> To add those configs, need to follow prefix "ranger.service.config.param" + 
> config parameter.
> 
> For blueprint based installs need to give those configs as below
> "ranger--plugin-properties": {
>   "ranger.service.config.param.configParameter1" : "vaule1",
>   "ranger.service.config.param.configParameter2" : "value2"
> }
> 
> For example in case of Hive
> "ranger-hive-plugin-properties": {
> "ranger.service.config.param.enable.hive.metastore.lookup": "false",
> "ranger.service.config.param.hive.site.file.path": 
> "/etc/hive/conf/hive-site.xml"
> }
> 
> For UI based installs add those properties in custom section under 
> ranger--plugin-properties. So that first restart of the component 
> can take those value after enabling the plugin for the component.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
>  04a5bb1 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  21d172b 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
>  268d81c 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
>  41f44c3 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/params_linux.py
>  62a5edd 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  880ddc5 
>   
> ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/params_linux.py
>  880ddc5 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py
>  b338add 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py
>  8beb2c0 
>   
> ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py
>  db59973 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py
>  5d8a5f3 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  03d3fb2 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py
>  d069722 
> 
> 
> Diff: https://reviews.apache.org/r/58106/diff/1/
> 
> 
> Testing
> ---
> 
> Tested Enabling of plugins by adding custom configs.
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 58002: AMBARI-20592 ServicePresences check should also check for services dropped from a release

[Alejandro Fernandez]

> On March 28, 2017, 9:01 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/checks/ServicePresenceCheck.java
> > Lines 161 (patched)
> > 
> >
> > Are these all removed services, or only the ones that participate in 
> > the orchestration?
> > 
> > What happens if a service, say AMS or SmartSense is dropped from the 
> > stack, doesn't participate in EU/RU?
> > 
> > Then the PreChecks shouldn't fail because of it.
> 
> Di Li wrote:
> Hello Alejandro,
> 
> I want all services to be checked, because when a service is dropped from 
> a stack, it's gone and doesn't matter if it;s included in the upgrade 
> orchestration.

I'm asking if we need to filter out for only services that participate in the 
orchestration


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58002/#review170340
---


On March 28, 2017, 8:04 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58002/
> ---
> 
> (Updated March 28, 2017, 8:04 p.m.)
> 
> 
> Review request for Ambari and Tim Thorpe.
> 
> 
> Bugs: AMBARI-20592
> https://issues.apache.org/jira/browse/AMBARI-20592
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ServicePresence check currently checks the following two cases
> 1. services that do not support upgrade, and must be reinstalled post EU/RU
> 2. services replaced by other services in the new release, where the old 
> service must be dropped before EU/RU, and the replacement can be added post 
> the stack upgrade.
> The JIRA is to add one more use case to the check, that;s to check for 
> services dropped from the new release (end of life)
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java
>  71c5857 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicePresenceCheck.java
>  4642b88 
>   
> ambari-server/src/test/java/org/apache/ambari/server/checks/ServicePresenceCheckTest.java
>  03b0e81 
> 
> 
> Diff: https://reviews.apache.org/r/58002/diff/1/
> 
> 
> Testing
> ---
> 
> unit tests
> patch Ambari trunk cluster, update upgrade.xml to include for example YARN as 
> a removed service, run EU/RU, verify precheck fails and couldn't proceed 
> (expected)
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 58109: Large Long Running Requests Can Slow Down the ActionScheduler

[Jonathan Hurley]

> On March 31, 2017, 5:07 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java
> > Lines 184-185 (original), 153-154 (patched)
> > 
> >
> > By using a more comple query, could we avoid making multiple calls the 
> > the DB to get the stage entities?
> > 
> > The following (non-JPA) query should do the trick once properly 
> > formatted for JPA. However, I am not sure if all DBs would support it.  
> > Apparenly PostgreSQL does, according to my test, and I know ath MySQL does. 
> >  I am not sure about other databases able to be used with Ambari. 
> > 
> > ```
> > SELECT *
> > FROM stage s 
> > INNER JOIN (
> >   SELECT s.request_id, MIN(s.stage_id) AS stage_id 
> >   FROM stage s 
> >   INNER JOIN host_role_command hrc ON (hrc.stage_id = s.stage_id AND 
> > hrc.request_id = s.request_id) 
> >   WHERE hrc.status  IN ('COMPLETED')
> >   GROUP BY s.request_id 
> >   ORDER BY s.request_id
> > ) AS foo ON (s.request_id = foo.request_id and s.stage_id = 
> > foo.stage_id); 
> > ```

This doesn't call into the database multiple times. The 2nd hit is a cache-only 
lookup. I think when I was researching how to do this, that query had problems 
on some databases... Namely; how do you get the entity from it when the 
request_id is in the returned results.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58109/#review170773
---


On March 31, 2017, 3:02 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58109/
> ---
> 
> (Updated March 31, 2017, 3:02 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.
> 
> 
> Bugs: BUG-20646
> https://issues.apache.org/jira/browse/BUG-20646
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When creating a massive request (a rolling upgrade on a cluster with 1000 
> nodes), the size of the request seems to slow down the {{ActionScheduler}}. 
> Each command was taking between 1 to 2 minutes to run (even server-side 
> tasks). 
> 
> The cause of this can be seen in the following two stack traces:
> 
> {code:title=ActionSchedulerImpl}
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
>   at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
>   at 
> org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:303)
>   at 
> org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:341)
>   at 
> org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> 
> {code:title=Server Action Executor}
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:700)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
>   at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
>   at 
> org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
>   at 
> org.apache.ambari.server.actionmanager.Request.(Request.java:199)
>   at 
> org.apache.ambari.server.actionmanager.Request$$FastClassByGuice$$9071e03.newInstance()
>   at 
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
>   at 
> com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
>   at 
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
>   at 
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
>   at 
> com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
>   at 
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
>   at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
>   at 
> com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
>   at com.sun.proxy.$Proxy26.createExisting(Unknown Source)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getRequests(ActionDBAccessorImpl.java:784)
>   at 
> 

Re: Review Request 58109: Large Long Running Requests Can Slow Down the ActionScheduler

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58109/#review170773
---


Ship it!




Ship It!


ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java
Lines 184-185 (original), 153-154 (patched)


By using a more comple query, could we avoid making multiple calls the the 
DB to get the stage entities?

The following (non-JPA) query should do the trick once properly formatted 
for JPA. However, I am not sure if all DBs would support it.  Apparenly 
PostgreSQL does, according to my test, and I know ath MySQL does.  I am not 
sure about other databases able to be used with Ambari. 

```
SELECT *
FROM stage s 
INNER JOIN (
  SELECT s.request_id, MIN(s.stage_id) AS stage_id 
  FROM stage s 
  INNER JOIN host_role_command hrc ON (hrc.stage_id = s.stage_id AND 
hrc.request_id = s.request_id) 
  WHERE hrc.status  IN ('COMPLETED')
  GROUP BY s.request_id 
  ORDER BY s.request_id
) AS foo ON (s.request_id = foo.request_id and s.stage_id = foo.stage_id); 
```


- Robert Levas


On March 31, 2017, 3:02 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58109/
> ---
> 
> (Updated March 31, 2017, 3:02 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.
> 
> 
> Bugs: BUG-20646
> https://issues.apache.org/jira/browse/BUG-20646
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When creating a massive request (a rolling upgrade on a cluster with 1000 
> nodes), the size of the request seems to slow down the {{ActionScheduler}}. 
> Each command was taking between 1 to 2 minutes to run (even server-side 
> tasks). 
> 
> The cause of this can be seen in the following two stack traces:
> 
> {code:title=ActionSchedulerImpl}
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
>   at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
>   at 
> org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:303)
>   at 
> org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:341)
>   at 
> org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> 
> {code:title=Server Action Executor}
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:700)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
>   at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
>   at 
> org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
>   at 
> org.apache.ambari.server.actionmanager.Request.(Request.java:199)
>   at 
> org.apache.ambari.server.actionmanager.Request$$FastClassByGuice$$9071e03.newInstance()
>   at 
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
>   at 
> com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
>   at 
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
>   at 
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
>   at 
> com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
>   at 
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
>   at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
>   at 
> com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
>   at com.sun.proxy.$Proxy26.createExisting(Unknown Source)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getRequests(ActionDBAccessorImpl.java:784)
>   at 
> org.apache.ambari.server.serveraction.ServerActionExecutor.cleanRequestShareDataContexts(ServerActionExecutor.java:259)
>   - locked <0x7ff0a14083c8> (a java.util.HashMap)
>   at 
> org.apache.ambari.server.serveraction.ServerActionExecutor.doWork(ServerActionExecutor.java:454)
>   at 
> org.apache.ambari.server.serveraction.ServerActionExecutor$1.run(ServerActionExecutor.java:160)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> 
> It's clear from these stacks that every 

Re: Review Request 58051: Misc fixes for ambari-server swagger integration prototype

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58051/#review170768
---


Ship it!




Ship It!

- Oliver Szabo


On March 29, 2017, 10:51 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58051/
> ---
> 
> (Updated March 29, 2017, 10:51 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Oliver Szabo, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20624
> https://issues.apache.org/jira/browse/AMBARI-20624
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of this ticket following changes are done:
> # swagger spec file will be committed to ambari project at path: 
> ambari-server/docs/api/generated/swagger.json
> # swagger doc file in html format will be auto-generated and committed at 
> path: ambari-server/docs/api/generated/index.html
> # ascidoc directory with md files will not be maintained
> # currently swagger spec file had some validation errors. these errors has 
> been addressed.
> 
> 
> Diffs
> -
> 
>   ambari-server/docs/api/asciidoc/definitions.adoc 76beb8c 
>   ambari-server/docs/api/asciidoc/overview.adoc e9e5649 
>   ambari-server/docs/api/asciidoc/paths.adoc d756051 
>   ambari-server/docs/api/generated/index.html PRE-CREATION 
>   ambari-server/docs/api/generated/swagger.json PRE-CREATION 
>   ambari-server/pom.xml 618d21f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/groups/GroupPrivilegeService.java
>  8a23885 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/groups/GroupService.java
>  ad9db28 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/groups/MemberService.java
>  ec02511 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/users/ActiveWidgetLayoutService.java
>  30714d4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/users/UserAuthorizationService.java
>  e6fda46 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/users/UserPrivilegeService.java
>  d6b4b29 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/users/UserService.java
>  ad9c485 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/views/ViewDataMigrationService.java
>  388f454 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/views/ViewInstanceService.java
>  9657c0e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/views/ViewPermissionService.java
>  cbc037b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/views/ViewPrivilegeService.java
>  e6d817f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/views/ViewService.java
>  1936d5b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/views/ViewVersionService.java
>  ed6ddb1 
> 
> 
> Diff: https://reviews.apache.org/r/58051/diff/1/
> 
> 
> Testing
> ---
> 
> verified that the swagger spec file has no validation error.
> verified that html doc file is generated as expected.
> 
> 
> Thanks,
> 
> Jaimin Jetly
> 
>

Re: Review Request 57945: Log Search Configuration API

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57945/#review170765
---




ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/templates/input.config-accumulo.json.j2
Lines 1 (patched)


I think you should include the changes in UpgradeCatalog30.java.. at least 
deleting *-logsearch-conf files. (other remaining upgrade changes can goes in 
with an another patch)


- Oliver Szabo


On March 31, 2017, 10:15 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57945/
> ---
> 
> (Updated March 31, 2017, 10:15 a.m.)
> 
> 
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
> 
> 
> Bugs: AMBARI-20578
> https://issues.apache.org/jira/browse/AMBARI-20578
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Log Search should store it's configurations (inputs, filters) by using an 
> API, which the user may implement as well to use their own way of storing the 
> configurations. By default for now we offer to store everything in ZooKeeper.
> 
> Also separated Log Feeder config handling from the main class.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-config-api/pom.xml PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/InputConfigMonitor.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass1.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass2.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactoryTest.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/NonLogSearchConfigClass.java
>  PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-config-api/src/test/resources/log4j.xml 
> PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-config-zookeeper/pom.xml PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigZK.java
>  PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 25e4306 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  a47c71f 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
>  8e70850 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
>  f93 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
>  3c80e50 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputS3File.java
>  26f1ddb 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java
>  73cf449 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SSLUtil.java
>  80b34e0 
>   ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log4j.xml 
> 7ef967c 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputFileTest.java
>  08aa564 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputManagerTest.java
>  368a930 
>   ambari-logsearch/ambari-logsearch-server/pom.xml 52bda8d 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/common/PropertiesHelper.java
>  73a43ad 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/ShipperConfigManager.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/ShipperConfigResource.java
>  PRE-CREATION 
>   

Re: Review Request 58079: Cleanup temporary files needed for downloading client configurations response

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58079/#review170759
---


Ship it!




I tested out the patch. Nice job.

- Robert Levas


On March 31, 2017, 11:57 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58079/
> ---
> 
> (Updated March 31, 2017, 11:57 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Balázs Bence Sári, Laszlo 
> Puskas, Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20596
> https://issues.apache.org/jira/browse/AMBARI-20596
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A tar.gz file is generated during client configuration download. This file is 
> not cleand up and its file permission is world-readable, but should be 
> owner-readable.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> fad14fd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  e98c062 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java
>  8f0c66f 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b2636ab 
> 
> 
> Diff: https://reviews.apache.org/r/58079/diff/2/
> 
> 
> Testing
> ---
> 
> modified existing unittest, tested client config download manually:
>  - created cluster with yarn
>  - downloaded client configuration
>  - checked file permission
> 
> [root@c6401 vagrant]# ls -al /var/lib/ambari-server/data/tmp/*.tar.gz
> -rw--- 1 root root 6834 Mar 31 11:53 
> /var/lib/ambari-server/data/tmp/YARN_CLIENT-configs.tar.gz
> 
> 
> existing tests: passed
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 58079: Cleanup temporary files needed for downloading client configurations response

[Attila Magyar]

> On March 31, 2017, 5:45 p.m., Robert Levas wrote:
> > ambari-common/src/main/python/resource_management/libraries/script/script.py
> > Lines 877 (patched)
> > 
> >
> > It is not a good idea to chmod 600 the directory set as the Ambari 
> > server's temporary directory.  This, by default is 
> > `/var/lib/ambari-server/data/tmp`, but may be changed using the 
> > `server.tmp.dir` property in `ambari-server.properties` file.  So someone 
> > could set this a `/var/tmp`.
> > 
> > Ideally the _correct_ permission was set on this directory when it was 
> > created.

The conf_tmp_dir is not the ambari temp directory, but a random subdirectory 
created under the ambari temp dir.

conf_tmp_dir = tempfile.mkdtemp(dir=self.get_tmp_dir()) # <- this creates a 
temp dir in ambar tmp


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58079/#review170747
---


On March 31, 2017, 3:57 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58079/
> ---
> 
> (Updated March 31, 2017, 3:57 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Balázs Bence Sári, Laszlo 
> Puskas, Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20596
> https://issues.apache.org/jira/browse/AMBARI-20596
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A tar.gz file is generated during client configuration download. This file is 
> not cleand up and its file permission is world-readable, but should be 
> owner-readable.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> fad14fd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  e98c062 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java
>  8f0c66f 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b2636ab 
> 
> 
> Diff: https://reviews.apache.org/r/58079/diff/2/
> 
> 
> Testing
> ---
> 
> modified existing unittest, tested client config download manually:
>  - created cluster with yarn
>  - downloaded client configuration
>  - checked file permission
> 
> [root@c6401 vagrant]# ls -al /var/lib/ambari-server/data/tmp/*.tar.gz
> -rw--- 1 root root 6834 Mar 31 11:53 
> /var/lib/ambari-server/data/tmp/YARN_CLIENT-configs.tar.gz
> 
> 
> existing tests: passed
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 58109: Large Long Running Requests Can Slow Down the ActionScheduler

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58109/
---

(Updated March 31, 2017, 3:02 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Changes
---

Fixing a minor issue with scheduling the cache timer


Bugs: BUG-20646
https://issues.apache.org/jira/browse/BUG-20646


Repository: ambari


Description
---

When creating a massive request (a rolling upgrade on a cluster with 1000 
nodes), the size of the request seems to slow down the {{ActionScheduler}}. 
Each command was taking between 1 to 2 minutes to run (even server-side tasks). 

The cause of this can be seen in the following two stack traces:

{code:title=ActionSchedulerImpl}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:303)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:341)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
at java.lang.Thread.run(Thread.java:745)
{code}

{code:title=Server Action Executor}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:700)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.Request.(Request.java:199)
at 
org.apache.ambari.server.actionmanager.Request$$FastClassByGuice$$9071e03.newInstance()
at 
com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at 
com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
at 
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at 
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at 
com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
at 
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
at 
com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
at com.sun.proxy.$Proxy26.createExisting(Unknown Source)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getRequests(ActionDBAccessorImpl.java:784)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.cleanRequestShareDataContexts(ServerActionExecutor.java:259)
- locked <0x7ff0a14083c8> (a java.util.HashMap)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.doWork(ServerActionExecutor.java:454)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor$1.run(ServerActionExecutor.java:160)
at java.lang.Thread.run(Thread.java:745)
{code}

It's clear from these stacks that every {{PENDING}} stage (roughly 15,000) were 
being loaded into memory every second (and their accompanying task as well). 
This makes no sense as these methods don't need all stages - just the _next_ 
stage. This is because all stages are synchronous within a single request.

The proposed solution is to fix the {{StageEntity.findByCommandStatuses}} call 
so it doesn't return every stage:
{code}
SELECT stage.requestid, 
   MIN(stage.stageid) 
FROM   stageentity stage, 
   hostrolecommandentity hrc 
WHERE  hrc.status IN :statuses 
   AND hrc.stageid = stage.stageid 
   AND hrc.requestid = stage.requestid 
GROUP  BY stage.requestid 
{code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessor.java
 9325d03 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
 ab4feaa 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java
 0984c5c 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java 
5151fb3 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java
 f68338f 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java
 b0be6b3 
  
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java
 

Re: Review Request 58079: Cleanup temporary files needed for downloading client configurations response

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58079/#review170747
---


Fix it, then Ship it!




Ship It!


ambari-common/src/main/python/resource_management/libraries/script/script.py
Lines 877 (patched)


It is not a good idea to chmod 600 the directory set as the Ambari server's 
temporary directory.  This, by default is `/var/lib/ambari-server/data/tmp`, 
but may be changed using the `server.tmp.dir` property in 
`ambari-server.properties` file.  So someone could set this a `/var/tmp`.

Ideally the _correct_ permission was set on this directory when it was 
created.


- Robert Levas


On March 31, 2017, 11:57 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58079/
> ---
> 
> (Updated March 31, 2017, 11:57 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Balázs Bence Sári, Laszlo 
> Puskas, Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20596
> https://issues.apache.org/jira/browse/AMBARI-20596
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A tar.gz file is generated during client configuration download. This file is 
> not cleand up and its file permission is world-readable, but should be 
> owner-readable.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> fad14fd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  e98c062 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java
>  8f0c66f 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b2636ab 
> 
> 
> Diff: https://reviews.apache.org/r/58079/diff/2/
> 
> 
> Testing
> ---
> 
> modified existing unittest, tested client config download manually:
>  - created cluster with yarn
>  - downloaded client configuration
>  - checked file permission
> 
> [root@c6401 vagrant]# ls -al /var/lib/ambari-server/data/tmp/*.tar.gz
> -rw--- 1 root root 6834 Mar 31 11:53 
> /var/lib/ambari-server/data/tmp/YARN_CLIENT-configs.tar.gz
> 
> 
> existing tests: passed
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 58109: Large Long Running Requests Can Slow Down the ActionScheduler

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58109/#review170746
---


Ship it!




Ship It!

- Alejandro Fernandez


On March 31, 2017, 5:22 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58109/
> ---
> 
> (Updated March 31, 2017, 5:22 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.
> 
> 
> Bugs: BUG-20646
> https://issues.apache.org/jira/browse/BUG-20646
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When creating a massive request (a rolling upgrade on a cluster with 1000 
> nodes), the size of the request seems to slow down the {{ActionScheduler}}. 
> Each command was taking between 1 to 2 minutes to run (even server-side 
> tasks). 
> 
> The cause of this can be seen in the following two stack traces:
> 
> {code:title=ActionSchedulerImpl}
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
>   at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
>   at 
> org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:303)
>   at 
> org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:341)
>   at 
> org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> 
> {code:title=Server Action Executor}
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:700)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
>   at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
>   at 
> org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
>   at 
> org.apache.ambari.server.actionmanager.Request.(Request.java:199)
>   at 
> org.apache.ambari.server.actionmanager.Request$$FastClassByGuice$$9071e03.newInstance()
>   at 
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
>   at 
> com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
>   at 
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
>   at 
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
>   at 
> com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
>   at 
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
>   at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
>   at 
> com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
>   at com.sun.proxy.$Proxy26.createExisting(Unknown Source)
>   at 
> org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getRequests(ActionDBAccessorImpl.java:784)
>   at 
> org.apache.ambari.server.serveraction.ServerActionExecutor.cleanRequestShareDataContexts(ServerActionExecutor.java:259)
>   - locked <0x7ff0a14083c8> (a java.util.HashMap)
>   at 
> org.apache.ambari.server.serveraction.ServerActionExecutor.doWork(ServerActionExecutor.java:454)
>   at 
> org.apache.ambari.server.serveraction.ServerActionExecutor$1.run(ServerActionExecutor.java:160)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> 
> It's clear from these stacks that every {{PENDING}} stage (roughly 15,000) 
> were being loaded into memory every second (and their accompanying task as 
> well). This makes no sense as these methods don't need all stages - just the 
> _next_ stage. This is because all stages are synchronous within a single 
> request.
> 
> The proposed solution is to fix the {{StageEntity.findByCommandStatuses}} 
> call so it doesn't return every stage:
> {code}
> SELECT stage.requestid, 
>MIN(stage.stageid) 
> FROM   stageentity stage, 
>hostrolecommandentity hrc 
> WHERE  hrc.status IN :statuses 
>AND hrc.stageid = stage.stageid 
>AND hrc.requestid = stage.requestid 
> GROUP  BY stage.requestid 
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessor.java
>  9325d03 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
>  ab4feaa 
>   
> 

Re: Review Request 58109: Large Long Running Requests Can Slow Down the ActionScheduler

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58109/
---

(Updated March 31, 2017, 1:22 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Bugs: BUG-20646
https://issues.apache.org/jira/browse/BUG-20646


Repository: ambari


Description
---

When creating a massive request (a rolling upgrade on a cluster with 1000 
nodes), the size of the request seems to slow down the {{ActionScheduler}}. 
Each command was taking between 1 to 2 minutes to run (even server-side tasks). 

The cause of this can be seen in the following two stack traces:

{code:title=ActionSchedulerImpl}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:303)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:341)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
at java.lang.Thread.run(Thread.java:745)
{code}

{code:title=Server Action Executor}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:700)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.Request.(Request.java:199)
at 
org.apache.ambari.server.actionmanager.Request$$FastClassByGuice$$9071e03.newInstance()
at 
com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at 
com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
at 
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at 
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at 
com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
at 
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
at 
com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
at com.sun.proxy.$Proxy26.createExisting(Unknown Source)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getRequests(ActionDBAccessorImpl.java:784)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.cleanRequestShareDataContexts(ServerActionExecutor.java:259)
- locked <0x7ff0a14083c8> (a java.util.HashMap)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.doWork(ServerActionExecutor.java:454)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor$1.run(ServerActionExecutor.java:160)
at java.lang.Thread.run(Thread.java:745)
{code}

It's clear from these stacks that every {{PENDING}} stage (roughly 15,000) were 
being loaded into memory every second (and their accompanying task as well). 
This makes no sense as these methods don't need all stages - just the _next_ 
stage. This is because all stages are synchronous within a single request.

The proposed solution is to fix the {{StageEntity.findByCommandStatuses}} call 
so it doesn't return every stage:
{code}
SELECT stage.requestid, 
   MIN(stage.stageid) 
FROM   stageentity stage, 
   hostrolecommandentity hrc 
WHERE  hrc.status IN :statuses 
   AND hrc.stageid = stage.stageid 
   AND hrc.requestid = stage.requestid 
GROUP  BY stage.requestid 
{code}


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessor.java
 9325d03 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
 ab4feaa 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java
 0984c5c 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java 
5151fb3 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java
 f68338f 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java
 b0be6b3 
  
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java
 81eef3b 
  

Review Request 58109: Large Long Running Requests Can Slow Down the ActionScheduler

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58109/
---

Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Bugs: BUG-20646
https://issues.apache.org/jira/browse/BUG-20646


Repository: ambari


Description
---

When creating a massive request (a rolling upgrade on a cluster with 1000 
nodes), the size of the request seems to slow down the {{ActionScheduler}}. 
Each command was taking between 1 to 2 minutes to run (even server-side tasks). 

The cause of this can be seen in the following two stack traces:

{code:title=ActionSchedulerImpl}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getStagesInProgress(ActionDBAccessorImpl.java:303)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:341)
at 
org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
at java.lang.Thread.run(Thread.java:745)
{code}

{code:title=Server Action Executor}
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:700)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getTasks(ActionDBAccessorImpl.java:84)
at org.apache.ambari.server.actionmanager.Stage.(Stage.java:157)
at 
org.apache.ambari.server.actionmanager.StageFactoryImpl.createExisting(StageFactoryImpl.java:72)
at 
org.apache.ambari.server.actionmanager.Request.(Request.java:199)
at 
org.apache.ambari.server.actionmanager.Request$$FastClassByGuice$$9071e03.newInstance()
at 
com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at 
com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
at 
com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at 
com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at 
com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
at 
com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
at 
com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
at com.sun.proxy.$Proxy26.createExisting(Unknown Source)
at 
org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.getRequests(ActionDBAccessorImpl.java:784)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.cleanRequestShareDataContexts(ServerActionExecutor.java:259)
- locked <0x7ff0a14083c8> (a java.util.HashMap)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor.doWork(ServerActionExecutor.java:454)
at 
org.apache.ambari.server.serveraction.ServerActionExecutor$1.run(ServerActionExecutor.java:160)
at java.lang.Thread.run(Thread.java:745)
{code}

It's clear from these stacks that every {{PENDING}} stage (roughly 15,000) were 
being loaded into memory every second (and their accompanying task as well). 
This makes no sense as these methods don't need all stages - just the _next_ 
stage. This is because all stages are synchronous within a single request.

The proposed solution is to fix the {{StageEntity.findByCommandStatuses}} call 
so it doesn't return every stage:
{code}
SELECT stage.requestid, 
   MIN(stage.stageid) 
FROM   stageentity stage, 
   hostrolecommandentity hrc 
WHERE  hrc.status IN :statuses 
   AND hrc.stageid = stage.stageid 
   AND hrc.requestid = stage.requestid 
GROUP  BY stage.requestid 
{code}


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessor.java
 9325d03 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
 ab4feaa 
  
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java
 0984c5c 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java 
5151fb3 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java
 f68338f 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java
 b0be6b3 
  
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java
 81eef3b 
  
ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java
 2b5d2f3 
  

Re: Review Request 58079: Cleanup temporary files needed for downloading client configurations response

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58079/#review170736
---


Ship it!




Ship It!

- Sebastian Toader


On March 31, 2017, 5:57 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58079/
> ---
> 
> (Updated March 31, 2017, 5:57 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Balázs Bence Sári, Laszlo 
> Puskas, Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20596
> https://issues.apache.org/jira/browse/AMBARI-20596
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A tar.gz file is generated during client configuration download. This file is 
> not cleand up and its file permission is world-readable, but should be 
> owner-readable.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> fad14fd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  e98c062 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java
>  8f0c66f 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b2636ab 
> 
> 
> Diff: https://reviews.apache.org/r/58079/diff/2/
> 
> 
> Testing
> ---
> 
> modified existing unittest, tested client config download manually:
>  - created cluster with yarn
>  - downloaded client configuration
>  - checked file permission
> 
> [root@c6401 vagrant]# ls -al /var/lib/ambari-server/data/tmp/*.tar.gz
> -rw--- 1 root root 6834 Mar 31 11:53 
> /var/lib/ambari-server/data/tmp/YARN_CLIENT-configs.tar.gz
> 
> 
> existing tests: passed
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 58002: AMBARI-20592 ServicePresences check should also check for services dropped from a release

[Di Li]

> On March 28, 2017, 9:01 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/checks/ServicePresenceCheck.java
> > Lines 161 (patched)
> > 
> >
> > Are these all removed services, or only the ones that participate in 
> > the orchestration?
> > 
> > What happens if a service, say AMS or SmartSense is dropped from the 
> > stack, doesn't participate in EU/RU?
> > 
> > Then the PreChecks shouldn't fail because of it.

Hello Alejandro,

I want all services to be checked, because when a service is dropped from a 
stack, it's gone and doesn't matter if it;s included in the upgrade 
orchestration.


- Di


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58002/#review170340
---


On March 28, 2017, 8:04 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58002/
> ---
> 
> (Updated March 28, 2017, 8:04 p.m.)
> 
> 
> Review request for Ambari and Tim Thorpe.
> 
> 
> Bugs: AMBARI-20592
> https://issues.apache.org/jira/browse/AMBARI-20592
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ServicePresence check currently checks the following two cases
> 1. services that do not support upgrade, and must be reinstalled post EU/RU
> 2. services replaced by other services in the new release, where the old 
> service must be dropped before EU/RU, and the replacement can be added post 
> the stack upgrade.
> The JIRA is to add one more use case to the check, that;s to check for 
> services dropped from the new release (end of life)
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java
>  71c5857 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServicePresenceCheck.java
>  4642b88 
>   
> ambari-server/src/test/java/org/apache/ambari/server/checks/ServicePresenceCheckTest.java
>  03b0e81 
> 
> 
> Diff: https://reviews.apache.org/r/58002/diff/1/
> 
> 
> Testing
> ---
> 
> unit tests
> patch Ambari trunk cluster, update upgrade.xml to include for example YARN as 
> a removed service, run EU/RU, verify precheck fails and couldn't proceed 
> (expected)
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 58079: Cleanup temporary files needed for downloading client configurations response

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58079/
---

(Updated March 31, 2017, 3:57 p.m.)


Review request for Ambari, Attila Doroszlai, Balázs Bence Sári, Laszlo Puskas, 
Robert Levas, and Sebastian Toader.


Changes
---

cleaning up command json files


Bugs: AMBARI-20596
https://issues.apache.org/jira/browse/AMBARI-20596


Repository: ambari


Description (updated)
---

A tar.gz file is generated during client configuration download. This file is 
not cleand up and its file permission is world-readable, but should be 
owner-readable.


Diffs (updated)
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
fad14fd 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 e98c062 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java
 8f0c66f 
  ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b2636ab 


Diff: https://reviews.apache.org/r/58079/diff/2/

Changes: https://reviews.apache.org/r/58079/diff/1-2/


Testing
---

modified existing unittest, tested client config download manually:
 - created cluster with yarn
 - downloaded client configuration
 - checked file permission

[root@c6401 vagrant]# ls -al /var/lib/ambari-server/data/tmp/*.tar.gz
-rw--- 1 root root 6834 Mar 31 11:53 
/var/lib/ambari-server/data/tmp/YARN_CLIENT-configs.tar.gz


existing tests: passed


Thanks,

Attila Magyar

Re: Review Request 58092: Upgrade server-side actions should be performed only one time per group (ncole)

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58092/#review170724
---


Ship it!




Just the nit about the logging. I agree about not adding the extra work for 
being able to say once|always when scoping a server-side task. I just don't see 
why it would ever need to run more than once. They, then, should be scoped 
implicitely to only run once.


ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ColocatedGrouping.java
Lines 245 (patched)


Can we make this LOG.debug("", tw) to prevent the toString() when DEBUG is 
off?


- Jonathan Hurley


On March 30, 2017, 9:01 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58092/
> ---
> 
> (Updated March 30, 2017, 9:01 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and 
> Jonathan Hurley.
> 
> 
> Bugs: AMBARI-20640
> https://issues.apache.org/jira/browse/AMBARI-20640
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Some server side actions, noteably config changes, should happen only one 
> time.  there are a couple of cases:
> 
> 
>   
> 
>class="org.apache.ambari.server.serveraction.upgrades.FixLzoCodecPath">
> Verifying LZO codec path for mapreduce
>   
> 
>   
> 
>   
> 
>id="hdp_2_5_0_0_add_spark2_yarn_shuffle"/>
> 
> 
> 
> 
> 
>   
> 
>   
> 
> These server-side actions are currently being scheduled for every instance of 
> MAPREDUCE2_CLIENT and NODEMANAGER.  Luckily, they're idempotent so no harm is 
> being done, but it is unncessary scheduling of tasks.
> 
> Originally I had intended to add another tag to the declaration to control 
> this behavior, but there actually isn't any use case that requires a 
> server-side action to happen for every host.  That effort has been shelved 
> for a later time if it ever becomes necessary.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ColocatedGrouping.java
>  c939320a14 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/TaskWrapperBuilder.java
>  bd2bf14d81 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  483880a1b1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
>  8e5ad0aadc 
>   
> ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_multi_server_tasks.xml
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/58092/diff/1/
> 
> 
> Testing
> ---
> 
> Manual.  Automated unit tests:
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 23:24.474s
> [INFO] Finished at: Thu Mar 30 18:52:54 EDT 2017
> [INFO] Final Memory: 31M/301M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 57945: Log Search Configuration API

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57945/#review170721
---


Ship it!




Patch looks fine, just a minor suggestion for a future patch in this area.


ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
Lines 38 (patched)


Thanks for adding this.

Just as a comment, it might be good to have LogSearchConfig extend the 
java.lang.Closeable interface, but that shouldn't hold up the patch.  

It might just be better in the long term to use the JDK interface, but the 
close() method here is fine as well.

If java.lang.Closeable is extended in this interface, then it opens up the 
possibility of using this config object within a "try-with-resources" block, 
which would simplify the cleanup, but again this is not required for the patch, 
just a suggestion.


- Robert Nettleton


On March 31, 2017, 10:15 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57945/
> ---
> 
> (Updated March 31, 2017, 10:15 a.m.)
> 
> 
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
> 
> 
> Bugs: AMBARI-20578
> https://issues.apache.org/jira/browse/AMBARI-20578
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Log Search should store it's configurations (inputs, filters) by using an 
> API, which the user may implement as well to use their own way of storing the 
> configurations. By default for now we offer to store everything in ZooKeeper.
> 
> Also separated Log Feeder config handling from the main class.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-config-api/pom.xml PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/InputConfigMonitor.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass1.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass2.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactoryTest.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/NonLogSearchConfigClass.java
>  PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-config-api/src/test/resources/log4j.xml 
> PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-config-zookeeper/pom.xml PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigZK.java
>  PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 25e4306 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  a47c71f 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
>  8e70850 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
>  f93 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
>  3c80e50 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputS3File.java
>  26f1ddb 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java
>  73cf449 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SSLUtil.java
>  80b34e0 
>   ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log4j.xml 
> 7ef967c 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputFileTest.java
>  08aa564 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputManagerTest.java
>  368a930 
>  

Review Request 58107: Integrate coilmq stomp server as a mock server for ambari-agent unittests.

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58107/
---

Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.


Bugs: AMBARI-20645
https://issues.apache.org/jira/browse/AMBARI-20645


Repository: ambari


Description
---

Coilmq is an apache licensed product   
Which provides a pretty simple python stomp server api.

Also the code includes the BaseStompServerTestCase.py which is a base class
for the ambari-agent tests which want to use stomp tests with ease.

The setup of every test method the test starts coilmq server and on every test
method tearodown it shutdowns it.


Diffs
-

  NOTICE.txt 3b65239 
  ambari-agent/src/test/python/ambari_agent/BaseStompServerTestCase.py 
PRE-CREATION 
  ambari-agent/src/test/python/ambari_agent/TestAgentStompResponses.py 
PRE-CREATION 
  ambari-common/src/test/python/coilmq/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/auth/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/auth/simple.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/config/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/config/coilmq.cfg-sample PRE-CREATION 
  ambari-common/src/test/python/coilmq/config/defaults.cfg PRE-CREATION 
  ambari-common/src/test/python/coilmq/engine.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/exception.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/protocol/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/queue.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/scheduler.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/server/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/server/socket_server.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/start.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/store/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/store/dbm.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/store/memory.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/store/rds.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/store/sa/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/store/sa/meta.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/store/sa/model.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/topic.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/util/__init__.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/util/concurrency.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/util/frames.py PRE-CREATION 
  ambari-common/src/test/python/coilmq/util/six.py PRE-CREATION 
  pom.xml 6f9bfdd 


Diff: https://reviews.apache.org/r/58107/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 58100: Remove configuring JDBC dependencies that are getting configured for Zeppelin interpreter

[Prabhjyot Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58100/
---

(Updated March 31, 2017, 8:02 p.m.)


Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, DIPAYAN 
BHOWMICK, Jayush Luniya, Prabhjyot Singh, Rohit Choudhary, Renjith Kamath, and 
Sumit Mohanty.


Bugs: AMBARI-20643
https://issues.apache.org/jira/browse/AMBARI-20643


Repository: ambari


Description
---

Remove configuring JDBC dependencies that are getting configured for Zeppelin 
interpreter as it causes more confusion than the resolution.


Diffs
-

  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
 2b21f8fe1f 


Diff: https://reviews.apache.org/r/58100/diff/1/


Testing
---

Manuaully tested on Ubuntu 14.04


Thanks,

Prabhjyot Singh

Re: Review Request 58105: Ambari Log Search Web - fix some build npm errors

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58105/#review170716
---


Ship it!




Ship It!

- Robert Nettleton


On March 31, 2017, 11:21 a.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58105/
> ---
> 
> (Updated March 31, 2017, 11:21 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Aleksandr Kovalenko, Miklos 
> Gergely, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-20644
> https://issues.apache.org/jira/browse/AMBARI-20644
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently we have some build errors with npm:
> [ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 No repository field.
> [ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 No README data
> [ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 license should be a 
> valid SPDX license expression
> [ERROR] npm WARN deprecated gulp-minify-css@1.2.4: Please use gulp-clean-css
> [ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
> [ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before 
> will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as 
> soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
> [ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 
> or higher to avoid a RegExp DoS issue
> 
> Most of them can be fixed, by editing package.json properly (I skiped the 
> nested dependencies as grunt use some of them), so the result will be after 
> this patch:
> 
> [ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before 
> will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as 
> soon as possible. Use 'npm ls 
> graceful-fs' to find it in the tree.
> [ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 
> or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-web/package.json 9dad560 
> 
> 
> Diff: https://reviews.apache.org/r/58105/diff/1/
> 
> 
> Testing
> ---
> 
> done.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 58105: Ambari Log Search Web - fix some build npm errors

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58105/#review170713
---


Ship it!




Ship It!

- Miklos Gergely


On March 31, 2017, 11:21 a.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58105/
> ---
> 
> (Updated March 31, 2017, 11:21 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Aleksandr Kovalenko, Miklos 
> Gergely, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-20644
> https://issues.apache.org/jira/browse/AMBARI-20644
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently we have some build errors with npm:
> [ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 No repository field.
> [ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 No README data
> [ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 license should be a 
> valid SPDX license expression
> [ERROR] npm WARN deprecated gulp-minify-css@1.2.4: Please use gulp-clean-css
> [ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
> [ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before 
> will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as 
> soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
> [ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 
> or higher to avoid a RegExp DoS issue
> 
> Most of them can be fixed, by editing package.json properly (I skiped the 
> nested dependencies as grunt use some of them), so the result will be after 
> this patch:
> 
> [ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before 
> will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as 
> soon as possible. Use 'npm ls 
> graceful-fs' to find it in the tree.
> [ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 
> or higher to avoid a RegExp DoS issue
> [ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch 
> 3.0.2 or higher to avoid a RegExp DoS issue
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-web/package.json 9dad560 
> 
> 
> Diff: https://reviews.apache.org/r/58105/diff/1/
> 
> 
> Testing
> ---
> 
> done.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 58079: Add stronger file permissions to temporary files generated by downloading client configurations

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58079/#review170711
---


Ship it!




Ship It!

- Sebastian Toader


On March 31, 2017, 1:57 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58079/
> ---
> 
> (Updated March 31, 2017, 1:57 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Balázs Bence Sári, Laszlo 
> Puskas, Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20596
> https://issues.apache.org/jira/browse/AMBARI-20596
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A tar.gz file is generated during client configuration download. This file is 
> not cleand up and its file permission is world-readable, which may be a 
> security risk. I changed the permission to owner-readable.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> fad14fd 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b2636ab 
> 
> 
> Diff: https://reviews.apache.org/r/58079/diff/1/
> 
> 
> Testing
> ---
> 
> modified existing unittest, tested client config download manually:
>  - created cluster with yarn
>  - downloaded client configuration
>  - checked file permission
> 
> [root@c6401 vagrant]# ls -al /var/lib/ambari-server/data/tmp/*.tar.gz
> -rw--- 1 root root 6834 Mar 31 11:53 
> /var/lib/ambari-server/data/tmp/YARN_CLIENT-configs.tar.gz
> 
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Review Request 58079: Add stronger file permissions to temporary files generated by downloading client configurations

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58079/
---

Review request for Ambari, Attila Doroszlai, Balázs Bence Sári, Laszlo Puskas, 
Robert Levas, and Sebastian Toader.


Bugs: AMBARI-20596
https://issues.apache.org/jira/browse/AMBARI-20596


Repository: ambari


Description
---

A tar.gz file is generated during client configuration download. This file is 
not cleand up and its file permission is world-readable, which may be a 
security risk. I changed the permission to owner-readable.


Diffs
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
fad14fd 
  ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b2636ab 


Diff: https://reviews.apache.org/r/58079/diff/1/


Testing
---

modified existing unittest, tested client config download manually:
 - created cluster with yarn
 - downloaded client configuration
 - checked file permission

[root@c6401 vagrant]# ls -al /var/lib/ambari-server/data/tmp/*.tar.gz
-rw--- 1 root root 6834 Mar 31 11:53 
/var/lib/ambari-server/data/tmp/YARN_CLIENT-configs.tar.gz


existing tests: pending


Thanks,

Attila Magyar

Review Request 58105: Ambari Log Search Web - fix some build npm errors

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58105/
---

Review request for Ambari, Aleksandr Kovalenko, Miklos Gergely, and Robert 
Nettleton.


Bugs: AMBARI-20644
https://issues.apache.org/jira/browse/AMBARI-20644


Repository: ambari


Description
---

Currently we have some build errors with npm:
[ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 No repository field.
[ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 No README data
[ERROR] npm WARN package.json ambari-logsearch-web@0.5.0 license should be a 
valid SPDX license expression
[ERROR] npm WARN deprecated gulp-minify-css@1.2.4: Please use gulp-clean-css
[ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue
[ERROR] npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
[ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before 
will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon 
as possible. Use 'npm ls graceful-fs' to find it in the tree.
[ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue
[ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue

Most of them can be fixed, by editing package.json properly (I skiped the 
nested dependencies as grunt use some of them), so the result will be after 
this patch:

[ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue
[ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before 
will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon 
as possible. Use 'npm ls 
graceful-fs' to find it in the tree.
[ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue
[ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue


Diffs
-

  ambari-logsearch/ambari-logsearch-web/package.json 9dad560 


Diff: https://reviews.apache.org/r/58105/diff/1/


Testing
---

done.


Thanks,

Oliver Szabo

Re: Review Request 57945: Log Search Configuration API

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57945/
---

(Updated March 31, 2017, 10:15 a.m.)


Review request for Ambari, Oliver Szabo and Robert Nettleton.


Changes
---

Added close called by the shutdownhook for the config


Bugs: AMBARI-20578
https://issues.apache.org/jira/browse/AMBARI-20578


Repository: ambari


Description
---

Log Search should store it's configurations (inputs, filters) by using an API, 
which the user may implement as well to use their own way of storing the 
configurations. By default for now we offer to store everything in ZooKeeper.

Also separated Log Feeder config handling from the main class.


Diffs (updated)
-

  ambari-logsearch/ambari-logsearch-config-api/pom.xml PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/InputConfigMonitor.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass1.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass2.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactoryTest.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/NonLogSearchConfigClass.java
 PRE-CREATION 
  ambari-logsearch/ambari-logsearch-config-api/src/test/resources/log4j.xml 
PRE-CREATION 
  ambari-logsearch/ambari-logsearch-config-zookeeper/pom.xml PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigZK.java
 PRE-CREATION 
  ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 25e4306 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
 a47c71f 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
 8e70850 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
 f93 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
 3c80e50 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputS3File.java
 26f1ddb 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederUtil.java
 73cf449 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SSLUtil.java
 80b34e0 
  ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/log4j.xml 
7ef967c 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputFileTest.java
 08aa564 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/InputManagerTest.java
 368a930 
  ambari-logsearch/ambari-logsearch-server/pom.xml 52bda8d 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/common/PropertiesHelper.java
 73a43ad 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/ShipperConfigManager.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/ShipperConfigResource.java
 PRE-CREATION 
  ambari-logsearch/ambari-logsearch-web/.gitignore PRE-CREATION 
  ambari-logsearch/pom.xml 1e63ced 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/configuration/accumulo-logsearch-conf.xml
 ff4f695 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/templates/input.config-accumulo.json.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-logsearch-conf.xml
 1de8c46 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/templates/input.config-ambari-infra.json.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-logsearch-conf.xml
 72d44db 
  

Re: Review Request 58100: Remove configuring JDBC dependencies that are getting configured for Zeppelin interpreter

[Renjith Kamath]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58100/#review170686
---


Ship it!




Ship It!

- Renjith Kamath


On March 31, 2017, 6:57 a.m., Prabhjyot Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58100/
> ---
> 
> (Updated March 31, 2017, 6:57 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, DIPAYAN 
> BHOWMICK, Jayush Luniya, Prabhjyot Singh, Rohit Choudhary, Renjith Kamath, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20643
> https://issues.apache.org/jira/browse/AMBARI-20643
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Remove configuring JDBC dependencies that are getting configured for Zeppelin 
> interpreter as it causes more confusion than the resolution.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2b21f8fe1f 
> 
> 
> Diff: https://reviews.apache.org/r/58100/diff/1/
> 
> 
> Testing
> ---
> 
> Manuaully tested on Ubuntu 14.04
> 
> 
> Thanks,
> 
> Prabhjyot Singh
> 
>

Re: Review Request 57959: Add (optional) master_kdcs to kerberos-env and generated krb5.conf file

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57959/#review170685
---


Ship it!




Ship It!

- Sebastian Toader


On March 30, 2017, 9:04 p.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57959/
> ---
> 
> (Updated March 30, 2017, 9:04 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Laszlo Puskas, Robert Levas, Sandor 
> Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20586
> https://issues.apache.org/jira/browse/AMBARI-20586
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added master_kdc to kerberos env and the krb5 config template.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  25f2001 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/configuration/kerberos-env.xml
>  965ee13 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/configuration/krb5-conf.xml
>  c692b92 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  8cb1f0c 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/params.py
>  57f062a 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/templates/krb5_conf.j2
>  0191953 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/configuration/kerberos-env.xml
>  7016437 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/configuration/krb5-conf.xml
>  c692b92 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  70c81e7 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/params.py
>  3533b35 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/templates/krb5_conf.j2
>  0191953 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  fa2082b 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> 6727b2f 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/use_cases.py 17d9f88 
> 
> 
> Diff: https://reviews.apache.org/r/57959/diff/4/
> 
> 
> Testing
> ---
> 
> - Tested manually
> - Updated Java unit tests
> - All ambari-server unit tests pass.
> 
> 
> Thanks,
> 
> Balázs Bence Sári
> 
>

Re: Review Request 58083: AMBARI-20625. Hadoop QA javac warnings increase check is broken -- fix for ambari-server

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58083/#review170684
---


Ship it!




Ship It!

- Sebastian Toader


On March 30, 2017, 6 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58083/
> ---
> 
> (Updated March 30, 2017, 6 p.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Sandor Magyari, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-20625
> https://issues.apache.org/jira/browse/AMBARI-20625
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added `-Xlint:all` flag.  Currently this has effect only on `ambari-server` 
> module.
> 
> 
> Diffs
> -
> 
>   dev-support/test-patch.sh 661f871c6cce927688f17d1aa77aba37eae9bbe6 
> 
> 
> Diff: https://reviews.apache.org/r/58083/diff/1/
> 
> 
> Testing
> ---
> 
> Tested with patch that increased the number of warnings, got:
> 
> ```
>   {color:red}-1 javac{color}.  The applied patch generated 496 javac 
> compiler warnings (more than the trunk's current 494 warnings).
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 58100: Remove configuring JDBC dependencies that are getting configured for Zeppelin interpreter

[Gaurav Nagar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58100/#review170683
---


Ship it!




Ship It!

- Gaurav Nagar


On March 31, 2017, 6:57 a.m., Prabhjyot Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58100/
> ---
> 
> (Updated March 31, 2017, 6:57 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, DIPAYAN 
> BHOWMICK, Jayush Luniya, Prabhjyot Singh, Rohit Choudhary, Renjith Kamath, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20643
> https://issues.apache.org/jira/browse/AMBARI-20643
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Remove configuring JDBC dependencies that are getting configured for Zeppelin 
> interpreter as it causes more confusion than the resolution.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2b21f8fe1f 
> 
> 
> Diff: https://reviews.apache.org/r/58100/diff/1/
> 
> 
> Testing
> ---
> 
> Manuaully tested on Ubuntu 14.04
> 
> 
> Thanks,
> 
> Prabhjyot Singh
> 
>

Re: Review Request 58100: Remove configuring JDBC dependencies that are getting configured for Zeppelin interpreter

[Pallav Kulshreshtha]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58100/#review170682
---


Ship it!




Ship It!

- Pallav Kulshreshtha


On March 31, 2017, 6:57 a.m., Prabhjyot Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58100/
> ---
> 
> (Updated March 31, 2017, 6:57 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, DIPAYAN 
> BHOWMICK, Jayush Luniya, Prabhjyot Singh, Rohit Choudhary, Renjith Kamath, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20643
> https://issues.apache.org/jira/browse/AMBARI-20643
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Remove configuring JDBC dependencies that are getting configured for Zeppelin 
> interpreter as it causes more confusion than the resolution.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2b21f8fe1f 
> 
> 
> Diff: https://reviews.apache.org/r/58100/diff/1/
> 
> 
> Testing
> ---
> 
> Manuaully tested on Ubuntu 14.04
> 
> 
> Thanks,
> 
> Prabhjyot Singh
> 
>

Re: Review Request 58100: Remove configuring JDBC dependencies that are getting configured for Zeppelin interpreter

[Rohit Choudhary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58100/#review170681
---


Ship it!




Ship It!

- Rohit Choudhary


On March 31, 2017, 6:57 a.m., Prabhjyot Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58100/
> ---
> 
> (Updated March 31, 2017, 6:57 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, DIPAYAN 
> BHOWMICK, Jayush Luniya, Prabhjyot Singh, Rohit Choudhary, Renjith Kamath, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20643
> https://issues.apache.org/jira/browse/AMBARI-20643
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Remove configuring JDBC dependencies that are getting configured for Zeppelin 
> interpreter as it causes more confusion than the resolution.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2b21f8fe1f 
> 
> 
> Diff: https://reviews.apache.org/r/58100/diff/1/
> 
> 
> Testing
> ---
> 
> Manuaully tested on Ubuntu 14.04
> 
> 
> Thanks,
> 
> Prabhjyot Singh
> 
>

Review Request 58100: Remove configuring JDBC dependencies that are getting configured for Zeppelin interpreter

[Prabhjyot Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58100/
---

Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, DIPAYAN 
BHOWMICK, Jayush Luniya, Prabhjyot Singh, Rohit Choudhary, Renjith Kamath, and 
Sumit Mohanty.


Bugs: AMBARI-20643
https://issues.apache.org/jira/browse/AMBARI-20643


Repository: ambari


Description
---

Remove configuring JDBC dependencies that are getting configured for Zeppelin 
interpreter as it causes more confusion than the resolution.


Diffs
-

  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
 2b21f8fe1f 


Diff: https://reviews.apache.org/r/58100/diff/1/


Testing
---

Manuaully tested on Ubuntu 14.04


Thanks,

Prabhjyot Singh