Re: Review Request 63805: Make Ambari configuration API consistent with existing API.

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63805/#review191028
---


Ship it!




Ship It!

- Nate Cole


On Nov. 14, 2017, 2:47 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63805/
> ---
> 
> (Updated Nov. 14, 2017, 2:47 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Jonathan Hurley, Nate Cole, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22418
> https://issues.apache.org/jira/browse/AMBARI-22418
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Make Ambari configuration API consistent with existing API. 
> 
> The current API entry point (as of AMBARI-21307) is 
> {{/api/v1/ambariconfigs}}. This should be more inline with the existing entry 
> point for Ambari server related data...  
> {{/api/v1/services/AMBARI/components/AMBARI_SERVER}}.  
> 
> The new API entry point for Ambari server related configuration data should 
> be {{/api/v1/services/AMBARI/components/AMBARI_SERVER/configurations}}.
> 
> See https://issues.apache.org/jira/browse/AMBARI-22418 for details about the 
> new entryoint.
> 
> *NOTE*: This patch contains a fix for a stack advisor issue that was 
> introduce in the patch for https://issues.apache.org/jira/browse/AMBARI-22347.
> 
> *NOTE*: Many changes are due to moving `RootServiceResponseFactory.Services` 
> from an inner class to `RootService` and 
> `RootServiceResponseFactory.Components` from an inner class to 
> `RootComponent`.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
>  f5fb6e9900 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/RootServiceComponentResourceDefinition.java
>  e8cb570c56 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariConfigurationRequestSwagger.java
>  2dca9f55f4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariConfigurationResponseSwagger.java
>  c55ac1dd60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariConfigurationService.java
>  86ed666f41 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  46ee65abea 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/RootServiceComponentConfigurationService.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/RootServiceService.java
>  5afb7dc90c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ldap/AmbariConfiguration.java
>  e9f0b1e1a4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ldap/LdapConfigurationRequest.java
>  2e478c4329 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ldap/LdapConfigurationService.java
>  00c13f6cfa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  91edf867ae 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
>  1425e1bd32 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/MaintenanceStateHelper.java
>  d9a585f630 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ResourceProviderFactory.java
>  711ae10f7e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/RootComponent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/RootService.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/RootServiceResponseFactory.java
>  ad9ed202dc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
>  1dc0841d19 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AmbariConfigurationResourceProvider.java
>  a24400dffd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RootServiceComponentConfigurationResourceProvider.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RootServiceComponentPropertyProvider.java
>  433c1fae90 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RootServiceComponentResourceProvider.java
>  3925aebdae 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java
>  78353735ac 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java
>  266c7e81c1 
>   
> 

Review Request 63822: Oozie Jobs on Hive Fail With Missing Tarball

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63822/
---

Review request for Ambari and Jonathan Hurley.


Repository: ambari


Description
---

STR:
- Install an HDP 2.6.3.0 cluster with Oozie, HIve, MapR, Yarn, ZK, HDFS
- Patch only Oozie to a new version, such as 2.6.3.1-1
- Attempt to run an Oozie job on Hive

The workflow job will fail with:
{code}
org.apache.oozie.action.ActionExecutorException: JA008: File does not exist: 
hdfs://jhurley-pu-1.openstacklocal:8020/hdp/apps/2.6.3.1-1/mapreduce/mapreduce.tar.gz
at 
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:457)
at 
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:441)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1211)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1382)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}

This is because Oozie appears to be trying to use a version of the MapReduce 
tarball which matches its version. However, since it does not ship its version, 
it cannot find it.


Diffs
-

  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
 68e6bf806c 


Diff: https://reviews.apache.org/r/63822/diff/1/


Testing
---

Manual.  Follow STR, then ran an oozie-on-hive job to confirm results.


No automated tests, as they exist already.


Thanks,

Nate Cole

Re: Review Request 63807: Add Native Libraries To Tez Tarball

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191022
---


Ship it!




Ship It!

- Nate Cole


On Nov. 14, 2017, 4:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 4:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63819: Ambari Infra, AMS, LogSearch are missing from list of services to be installed

[Madhuvanthi Radhakrishnan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63819/#review191017
---


Ship it!




Ship It!

- Madhuvanthi Radhakrishnan


On Nov. 14, 2017, 11:43 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63819/
> ---
> 
> (Updated Nov. 14, 2017, 11:43 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.
> 
> 
> Bugs: AMBARI-22447
> https://issues.apache.org/jira/browse/AMBARI-22447
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari Infra service is missing from list of services to be installed, This 
> is one of the necessary services for Atlas and Ranger, other-wise need to 
> setup Solr manually.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/3.0/services/AMBARI_INFRA/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/3.0/services/AMBARI_METRICS/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/3.0/services/LOGSEARCH/metainfo.xml
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63819/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Review Request 63819: Ambari Infra, AMS, LogSearch are missing from list of services to be installed

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63819/
---

Review request for Ambari and Jayush Luniya.


Bugs: AMBARI-22447
https://issues.apache.org/jira/browse/AMBARI-22447


Repository: ambari


Description
---

Ambari Infra service is missing from list of services to be installed, This is 
one of the necessary services for Atlas and Ranger, other-wise need to setup 
Solr manually.


Diffs
-

  
ambari-server/src/main/resources/stacks/HDP/3.0/services/AMBARI_INFRA/metainfo.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/3.0/services/AMBARI_METRICS/metainfo.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/3.0/services/LOGSEARCH/metainfo.xml 
PRE-CREATION 


Diff: https://reviews.apache.org/r/63819/diff/1/


Testing
---


Thanks,

Vitalyi Brodetskyi

Re: Review Request 63807: Add Native Libraries To Tez Tarball

[Dmytro Grinenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191009
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 54 (patched)


this is a bit dangerous, as would return the same function and calling it 
would make infinite recursion.

better replace to something like that, by skipping values which r not 
needed:

  _, mapreduce_source_file, _, _ = get_tarball_paths("mapreduce")
  _, tez_source_file, _, _ = get_tarball_paths("tez")
  
Round brackets do not needed here



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 73 (patched)


why we trying to copy directory, if it could be even not exists? 

SHoudk we fail at line 70 then?



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 219 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Line 261 (original), 316 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant


- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63807: Add Native Libraries To Tez Tarball

[Dmytro Grinenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191012
---



- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 63807: Add Native Libraries To Tez Tarball

[Dmytro Grinenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191009
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 54 (patched)


this is a bit dangerous, as would return the same function and calling it 
would make infinite recursion.

better replace to something like that, by skipping values which r not 
needed:

  _, mapreduce_source_file, _, _ = get_tarball_paths("mapreduce")
  _, tez_source_file, _, _ = get_tarball_paths("tez")
  
Round brackets do not needed here



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 73 (patched)


why we trying to copy directory, if it could be even not exists? 

SHoudk we fail at line 70 then?



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 219 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Line 261 (original), 316 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant


- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Review Request 63807: Add Native Libraries To Tez Tarball

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/
---

Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.


Bugs: AMBARI-22444
https://issues.apache.org/jira/browse/AMBARI-22444


Repository: ambari


Description
---

As part of the investigation for getting patch upgrades working in Ambari 
2.6.1, it was determined that the Tez tarball will need to have the hadoop 
native libraries added to it so that they can be detected from the tarball.

STR:
- Install ZK, MapR, Tez, Yarn, Hive
- Enable a non-LZO codec, like Snappy
- Patch Hive to a new version
- Change the following properties in {{tez-site}}:
-- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
-- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native

When Hive commands run, they will attempt to load the native snappy libraries 
from the Tez tarball and will fail with:
{code}
Caused by: java.io.IOException: Unable to get CompressorType for codec 
(org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing 
native libraries for the codec.
at 
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
{code}

In order to fix this, the Tez tarball should include the native hadoop 
libraries as well:
{code}
??? tez
?   ??? lib
?   ?   ??? native
?   ?   ?   ??? libhadoop.a
?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
?   ?   ?   ??? libhadoop.so.1.0.0
?   ?   ?   ??? libhadooppipes.a
?   ?   ?   ??? libhadooputils.a
?   ?   ?   ??? libhdfs.a
?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
?   ?   ?   ??? libhdfs.so.0.0.0
?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
?   ?   ?   ??? libsnappy.so.1.1.4
{code}


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 03b6213ac4 
  
ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
 c682c3e24a 


Diff: https://reviews.apache.org/r/63807/diff/1/


Testing
---

Manual testing to ensure the file is created and uploaded.

UNIT TESTS PENDING...


Thanks,

Jonathan Hurley

Re: Review Request 63800: AMBARI-22431 Able to add config type if EU/RU of the same stack (minor version upgrade)

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63800/
---

(Updated Nov. 14, 2017, 8:09 p.m.)


Review request for Ambari, Jonathan Hurley and Tim Thorpe.


Bugs: AMBARI-22431
https://issues.apache.org/jira/browse/AMBARI-22431


Repository: ambari


Description
---

EU/RU does not create/load default of new config types when the source stack 
and target stack is the same. This JIRA introduces a new config task type for 
creating/loading defaults of a new config type when stack does not change 
during EU/RU. This is particularly important for EU/RU of custom services where 
their stack may not change at all.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 66f5bf9 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureAction.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
 56f13ab 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 63d0993 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/CreateAndConfigureTask.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java
 6ab2fd2 
  ambari-server/src/main/resources/upgrade-pack.xsd 21606bd 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureActionTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63800/diff/2/

Changes: https://reviews.apache.org/r/63800/diff/1-2/


Testing
---

unit tests, patch Ambari 2.6 cluster with the changes, upgrade from HDP 2.6.2 
to 2.6.3, where zookeeper was modified to have zookeeper-fake that needs to be 
created and configured during EU.


File Attachments


config_type_tests_hdp_2.6
  
https://reviews.apache.org/media/uploaded/files/2017/11/14/06efe434-a4fa-4b74-bede-7293bb6e1bf5__config_type_tests_hdp_2.6.zip


Thanks,

Di Li

Review Request 63805: Make Ambari configuration API consistent with existing API.

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63805/
---

Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene Chekanskiy, 
Jonathan Hurley, Nate Cole, and Robert Nettleton.


Bugs: AMBARI-22418
https://issues.apache.org/jira/browse/AMBARI-22418


Repository: ambari


Description
---

Make Ambari configuration API consistent with existing API. 

The current API entry point (as of AMBARI-21307) is {{/api/v1/ambariconfigs}}. 
This should be more inline with the existing entry point for Ambari server 
related data...  {{/api/v1/services/AMBARI/components/AMBARI_SERVER}}.  

The new API entry point for Ambari server related configuration data should be 
{{/api/v1/services/AMBARI/components/AMBARI_SERVER/configurations}}.

See https://issues.apache.org/jira/browse/AMBARI-22418 for details about the 
new entryoint.

*NOTE*: This patch contains a fix for a stack advisor issue that was introduce 
in the patch for https://issues.apache.org/jira/browse/AMBARI-22347.

*NOTE*: Many changes are due to moving `RootServiceResponseFactory.Services` 
from an inner class to `RootService` and 
`RootServiceResponseFactory.Components` from an inner class to `RootComponent`.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
 f5fb6e9900 
  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/RootServiceComponentResourceDefinition.java
 e8cb570c56 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariConfigurationRequestSwagger.java
 2dca9f55f4 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariConfigurationResponseSwagger.java
 c55ac1dd60 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariConfigurationService.java
 86ed666f41 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 46ee65abea 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/RootServiceComponentConfigurationService.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/RootServiceService.java
 5afb7dc90c 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ldap/AmbariConfiguration.java
 e9f0b1e1a4 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ldap/LdapConfigurationRequest.java
 2e478c4329 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ldap/LdapConfigurationService.java
 00c13f6cfa 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 91edf867ae 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
 1425e1bd32 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/MaintenanceStateHelper.java
 d9a585f630 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/ResourceProviderFactory.java
 711ae10f7e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/RootComponent.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/RootService.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/RootServiceResponseFactory.java
 ad9ed202dc 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
 1dc0841d19 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AmbariConfigurationResourceProvider.java
 a24400dffd 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RootServiceComponentConfigurationResourceProvider.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RootServiceComponentPropertyProvider.java
 433c1fae90 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RootServiceComponentResourceProvider.java
 3925aebdae 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java
 78353735ac 
  
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java
 266c7e81c1 
  
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertStateChangedListener.java
 d5dc530394 
  
ambari-server/src/main/java/org/apache/ambari/server/ldap/domain/AmbariLdapConfiguration.java
 0c1ec0ae71 
  
ambari-server/src/main/java/org/apache/ambari/server/ldap/domain/AmbariLdapConfigurationFactory.java
 2b9f24be89 
  
ambari-server/src/main/java/org/apache/ambari/server/ldap/service/AmbariLdapConfigurationProvider.java
 b32d1ed9fb 
  
ambari-server/src/main/java/org/apache/ambari/server/metadata/AmbariServiceAlertDefinitions.java
 1e20571ab2 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/AlertDefinitionDAO.java
 cda03f307b 
  

Re: Review Request 63803: Log Feeder: do not try to connect AMS, when it is disabled

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63803/#review190990
---


Ship it!




Ship It!

- Robert Nettleton


On Nov. 14, 2017, 6:15 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63803/
> ---
> 
> (Updated Nov. 14, 2017, 6:15 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert 
> Nettleton.
> 
> 
> Bugs: AMBARI-22443
> https://issues.apache.org/jira/browse/AMBARI-22443
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Set ams client to null if any of their properties are null.
> 
> Its already merged/fixed in 2.6 branch
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
>  f446446 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
>  1094852 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetricsManagerTest.java
>  24042a7 
> 
> 
> Diff: https://reviews.apache.org/r/63803/diff/1/
> 
> 
> Testing
> ---
> 
> done
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 63803: Log Feeder: do not try to connect AMS, when it is disabled

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63803/#review190984
---


Ship it!




Ship It!

- Miklos Gergely


On Nov. 14, 2017, 6:15 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63803/
> ---
> 
> (Updated Nov. 14, 2017, 6:15 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert 
> Nettleton.
> 
> 
> Bugs: AMBARI-22443
> https://issues.apache.org/jira/browse/AMBARI-22443
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Set ams client to null if any of their properties are null.
> 
> Its already merged/fixed in 2.6 branch
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
>  f446446 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
>  1094852 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetricsManagerTest.java
>  24042a7 
> 
> 
> Diff: https://reviews.apache.org/r/63803/diff/1/
> 
> 
> Testing
> ---
> 
> done
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Review Request 63803: Log Feeder: do not try to connect AMS, when it is disabled

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63803/
---

Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Robert Nettleton.


Bugs: AMBARI-22443
https://issues.apache.org/jira/browse/AMBARI-22443


Repository: ambari


Description
---

Set ams client to null if any of their properties are null.

Its already merged/fixed in 2.6 branch


Diffs
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
 f446446 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
 1094852 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetricsManagerTest.java
 24042a7 


Diff: https://reviews.apache.org/r/63803/diff/1/


Testing
---

done


Thanks,

Oliver Szabo

Re: Review Request 63800: AMBARI-22431 Able to add config type if EU/RU of the same stack (minor version upgrade)

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63800/#review190968
---




ambari-server/src/main/resources/upgrade-pack.xsd
Lines 341 (patched)


Should this extend the configure task instead?


- Jonathan Hurley


On Nov. 14, 2017, 11:46 a.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63800/
> ---
> 
> (Updated Nov. 14, 2017, 11:46 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Tim Thorpe.
> 
> 
> Bugs: AMBARI-22431
> https://issues.apache.org/jira/browse/AMBARI-22431
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> EU/RU does not create/load default of new config types when the source stack 
> and target stack is the same. This JIRA introduces a new config task type for 
> creating/loading defaults of a new config type when stack does not change 
> during EU/RU. This is particularly important for EU/RU of custom services 
> where their stack may not change at all.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  66f5bf9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureAction.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
>  56f13ab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
>  63d0993 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/CreateAndConfigureTask.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java
>  6ab2fd2 
>   ambari-server/src/main/resources/upgrade-pack.xsd 21606bd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureActionTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63800/diff/1/
> 
> 
> Testing
> ---
> 
> unit tests, patch Ambari 2.6 cluster with the changes, upgrade from HDP 2.6.2 
> to 2.6.3, where zookeeper was modified to have zookeeper-fake that needs to 
> be created and configured during EU.
> 
> 
> File Attachments
> 
> 
> config_type_tests_hdp_2.6
>   
> https://reviews.apache.org/media/uploaded/files/2017/11/14/06efe434-a4fa-4b74-bede-7293bb6e1bf5__config_type_tests_hdp_2.6.zip
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 63787: AMBARI-22437 : Create an 'AD Manager' component in Ambari Metrics Service stack side.

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63787/
---

(Updated Nov. 14, 2017, 5:26 p.m.)


Review request for Ambari, Jayush Luniya and Sid Wagle.


Bugs: AMBARI-22437
https://issues.apache.org/jira/browse/AMBARI-22437


Repository: ambari


Description
---

Goals
Add a component called Anomaly-Detector in Ambari Metrics Service.
Create scripts to start/stop the component
Configuration management for Anomaly-Detector component.
Create Ambari stack scripts to manage lifecycle of the component - install, 
start, stop.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager.sh
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/log4j.properties
 PRE-CREATION 
  ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml cfa8124 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/resources/config.yml
 299a472 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala
 aa20223 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala
 28b2880 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/HBaseConfiguration.scala
 a7bbc66 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/MetricCollectorConfiguration.scala
 9418897 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/MetricDefinitionDBConfiguration.scala
 79a350c 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/LevelDbStoreAccessor.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/leveldb/LevelDBDatasource.scala
 6d185bf 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/ADMetadataProvider.scala
 3bcf4b0 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala
 ffa9944 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/AnomalyResource.scala
 c941ac3 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala
 aacea79 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/trend/TrendAnomalyInstance.scala
 125da34 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala
 104ccea 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/DefaultADResourceSpecTest.scala
 65cf609 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/leveldb/LevelDBDataSourceTest.scala
 2ddb7b8 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml
 489850f 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-env.xml
 99e93a6 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-log4j.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/metainfo.xml
 e80fae4 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
 4b960ff 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 a2bfe68 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/status_params.py
 0a92cd2 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/admanager_config.yaml.j2
 787aa3e 
  ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_admanager.py 
PRE-CREATION 
  ambari-server/src/test/python/stacks/2.0.6/configs/default.json 43c184c 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_ams_embedded.json 
eaaeeb5 


Diff: https://reviews.apache.org/r/63787/diff/2/

Changes: https://reviews.apache.org/r/63787/diff/1-2/


Testing
---

Manually Tested.
Relevant unit tests pass.


Thanks,

Aravindan Vijayan

Re: Review Request 63787: AMBARI-22437 : Create an 'AD Manager' component in Ambari Metrics Service stack side.

[Aravindan Vijayan]

> On Nov. 14, 2017, 6:39 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/leveldb/LevelDBDatasource.scala
> > Line 44 (original), 55 (patched)
> > 
> >
> > why get.. vs verb for behavior spec?

Without the 'get' prefix, I got the following error.

  * Unrecognized field at: metricDefinitionDB.verifyChecksums
Did you mean?:
  - dbDirPath


- Aravindan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63787/#review190921
---


On Nov. 14, 2017, 6:16 a.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63787/
> ---
> 
> (Updated Nov. 14, 2017, 6:16 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sid Wagle.
> 
> 
> Bugs: AMBARI-22437
> https://issues.apache.org/jira/browse/AMBARI-22437
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Goals
> Add a component called Anomaly-Detector in Ambari Metrics Service.
> Create scripts to start/stop the component
> Configuration management for Anomaly-Detector component.
> Create Ambari stack scripts to manage lifecycle of the component - install, 
> start, stop.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager.sh
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/log4j.properties
>  PRE-CREATION 
>   ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml cfa8124 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/resources/config.yml
>  299a472 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala
>  aa20223 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppModule.scala
>  28b2880 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/HBaseConfiguration.scala
>  a7bbc66 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/MetricCollectorConfiguration.scala
>  9418897 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/MetricDefinitionDBConfiguration.scala
>  79a350c 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/LevelDbStoreAccessor.scala
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/leveldb/LevelDBDatasource.scala
>  6d185bf 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/ADMetadataProvider.scala
>  3bcf4b0 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/metadata/MetricDefinitionServiceImpl.scala
>  ffa9944 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/AnomalyResource.scala
>  c941ac3 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/resource/MetricDefinitionResource.scala
>  aacea79 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/trend/TrendAnomalyInstance.scala
>  125da34 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala
>  104ccea 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/DefaultADResourceSpecTest.scala
>  65cf609 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/leveldb/LevelDBDataSourceTest.scala
>  2ddb7b8 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-config.xml
>  489850f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-env.xml
>  99e93a6 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-admanager-log4j.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/metainfo.xml
>  e80fae4 
>   
> 

Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63793/#review190965
---


Ship it!




Ship It!

- Robert Nettleton


On Nov. 14, 2017, 1:07 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63793/
> ---
> 
> (Updated Nov. 14, 2017, 1:07 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22395
> https://issues.apache.org/jira/browse/AMBARI-22395
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - added new config: "max_age_min" (similar as detach_time_min) to input 
> block. that stored in checkpoint files, during cleanup, based on that the 
> checkpoint file can be deleted.
> - based on "max_age_min" value, during bootstrap, its possible logfeeder wont 
> even monitor a file in the future.
> - created a new thread which periodically run the checkpoint cleanup (now in 
> every 2000 minutes ... its ok to run that rearly)
> - fix metrics sink, as its called if some properties are missing
> - add new input test files (testing wildcard pattern)
> - fix when fatal error thrown if no any files exist based on a wildcard 
> pattern
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
>  8548a20 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
>  96320e9 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
>  8e70850 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java
>  3910b9b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java
>  322a56d 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java
>  cc5d664 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
>  2d1bf40 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
>  1432c87 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java
>  843ae6b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java
>  8ee6d00 
>   ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a 
>   
> ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json
>  PRE-CREATION 
>   
> ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63793/diff/3/
> 
> 
> Testing
> ---
> 
> done, manually, added new input file for docker env.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 63800: AMBARI-22431 Able to add config type if EU/RU of the same stack (minor version upgrade)

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63800/#review190961
---


Ship it!




Ship It!

- Tim Thorpe


On Nov. 14, 2017, 4:46 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63800/
> ---
> 
> (Updated Nov. 14, 2017, 4:46 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Tim Thorpe.
> 
> 
> Bugs: AMBARI-22431
> https://issues.apache.org/jira/browse/AMBARI-22431
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> EU/RU does not create/load default of new config types when the source stack 
> and target stack is the same. This JIRA introduces a new config task type for 
> creating/loading defaults of a new config type when stack does not change 
> during EU/RU. This is particularly important for EU/RU of custom services 
> where their stack may not change at all.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  66f5bf9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureAction.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
>  56f13ab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
>  63d0993 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/CreateAndConfigureTask.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java
>  6ab2fd2 
>   ambari-server/src/main/resources/upgrade-pack.xsd 21606bd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureActionTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63800/diff/1/
> 
> 
> Testing
> ---
> 
> unit tests, patch Ambari 2.6 cluster with the changes, upgrade from HDP 2.6.2 
> to 2.6.3, where zookeeper was modified to have zookeeper-fake that needs to 
> be created and configured during EU.
> 
> 
> File Attachments
> 
> 
> config_type_tests_hdp_2.6
>   
> https://reviews.apache.org/media/uploaded/files/2017/11/14/06efe434-a4fa-4b74-bede-7293bb6e1bf5__config_type_tests_hdp_2.6.zip
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 63800: AMBARI-22431 Able to add config type if EU/RU of the same stack (minor version upgrade)

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63800/
---

(Updated Nov. 14, 2017, 4:46 p.m.)


Review request for Ambari, Jonathan Hurley and Tim Thorpe.


Bugs: AMBARI-22431
https://issues.apache.org/jira/browse/AMBARI-22431


Repository: ambari


Description
---

EU/RU does not create/load default of new config types when the source stack 
and target stack is the same. This JIRA introduces a new config task type for 
creating/loading defaults of a new config type when stack does not change 
during EU/RU. This is particularly important for EU/RU of custom services where 
their stack may not change at all.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 66f5bf9 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureAction.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
 56f13ab 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 63d0993 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/CreateAndConfigureTask.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java
 6ab2fd2 
  ambari-server/src/main/resources/upgrade-pack.xsd 21606bd 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureActionTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63800/diff/1/


Testing
---

unit tests, patch Ambari 2.6 cluster with the changes, upgrade from HDP 2.6.2 
to 2.6.3, where zookeeper was modified to have zookeeper-fake that needs to be 
created and configured during EU.


File Attachments (updated)


config_type_tests_hdp_2.6
  
https://reviews.apache.org/media/uploaded/files/2017/11/14/06efe434-a4fa-4b74-bede-7293bb6e1bf5__config_type_tests_hdp_2.6.zip


Thanks,

Di Li

Review Request 63800: AMBARI-22431 Able to add config type if EU/RU of the same stack (minor version upgrade)

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63800/
---

Review request for Ambari, Jonathan Hurley and Tim Thorpe.


Bugs: AMBARI-22431
https://issues.apache.org/jira/browse/AMBARI-22431


Repository: ambari


Description
---

EU/RU does not create/load default of new config types when the source stack 
and target stack is the same. This JIRA introduces a new config task type for 
creating/loading defaults of a new config type when stack does not change 
during EU/RU. This is particularly important for EU/RU of custom services where 
their stack may not change at all.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 66f5bf9 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureAction.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
 56f13ab 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 63d0993 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/CreateAndConfigureTask.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java
 6ab2fd2 
  ambari-server/src/main/resources/upgrade-pack.xsd 21606bd 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/CreateAndConfigureActionTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63800/diff/1/


Testing
---

unit tests, patch Ambari 2.6 cluster with the changes, upgrade from HDP 2.6.2 
to 2.6.3, where zookeeper was modified to have zookeeper-fake that needs to be 
created and configured during EU.


Thanks,

Di Li

Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63793/
---

(Updated Nov. 14, 2017, 1:07 p.m.)


Review request for Ambari, Krisztian Kasa and Robert Nettleton.


Changes
---

use waitIntervalMin as parameter


Bugs: AMBARI-22395
https://issues.apache.org/jira/browse/AMBARI-22395


Repository: ambari


Description
---

- added new config: "max_age_min" (similar as detach_time_min) to input block. 
that stored in checkpoint files, during cleanup, based on that the checkpoint 
file can be deleted.
- based on "max_age_min" value, during bootstrap, its possible logfeeder wont 
even monitor a file in the future.
- created a new thread which periodically run the checkpoint cleanup (now in 
every 2000 minutes ... its ok to run that rearly)
- fix metrics sink, as its called if some properties are missing
- add new input test files (testing wildcard pattern)
- fix when fatal error thrown if no any files exist based on a wildcard pattern


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
 8548a20 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
 96320e9 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
 8e70850 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java
 3910b9b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java
 322a56d 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java
 cc5d664 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
 2d1bf40 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
 1432c87 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java
 843ae6b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java
 8ee6d00 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a 
  
ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json
 PRE-CREATION 
  
ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63793/diff/3/

Changes: https://reviews.apache.org/r/63793/diff/2-3/


Testing
---

done, manually, added new input file for docker env.


Thanks,

Oliver Szabo

Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63793/#review190941
---


Ship it!




Ship It!

- Krisztian Kasa


On Nov. 14, 2017, 12:50 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63793/
> ---
> 
> (Updated Nov. 14, 2017, 12:50 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22395
> https://issues.apache.org/jira/browse/AMBARI-22395
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - added new config: "max_age_min" (similar as detach_time_min) to input 
> block. that stored in checkpoint files, during cleanup, based on that the 
> checkpoint file can be deleted.
> - based on "max_age_min" value, during bootstrap, its possible logfeeder wont 
> even monitor a file in the future.
> - created a new thread which periodically run the checkpoint cleanup (now in 
> every 2000 minutes ... its ok to run that rearly)
> - fix metrics sink, as its called if some properties are missing
> - add new input test files (testing wildcard pattern)
> - fix when fatal error thrown if no any files exist based on a wildcard 
> pattern
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
>  8548a20 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
>  96320e9 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
>  8e70850 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java
>  3910b9b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java
>  322a56d 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java
>  cc5d664 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
>  2d1bf40 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
>  1432c87 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java
>  843ae6b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java
>  8ee6d00 
>   ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a 
>   
> ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json
>  PRE-CREATION 
>   
> ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63793/diff/2/
> 
> 
> Testing
> ---
> 
> done, manually, added new input file for docker env.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63793/#review190940
---



waitInterval -> waitIntervalMin

- Krisztian Kasa


On Nov. 14, 2017, 12:50 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63793/
> ---
> 
> (Updated Nov. 14, 2017, 12:50 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22395
> https://issues.apache.org/jira/browse/AMBARI-22395
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - added new config: "max_age_min" (similar as detach_time_min) to input 
> block. that stored in checkpoint files, during cleanup, based on that the 
> checkpoint file can be deleted.
> - based on "max_age_min" value, during bootstrap, its possible logfeeder wont 
> even monitor a file in the future.
> - created a new thread which periodically run the checkpoint cleanup (now in 
> every 2000 minutes ... its ok to run that rearly)
> - fix metrics sink, as its called if some properties are missing
> - add new input test files (testing wildcard pattern)
> - fix when fatal error thrown if no any files exist based on a wildcard 
> pattern
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
>  8548a20 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
>  96320e9 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
>  8e70850 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java
>  3910b9b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java
>  322a56d 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java
>  cc5d664 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
>  2d1bf40 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
>  1432c87 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java
>  843ae6b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java
>  8ee6d00 
>   ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a 
>   
> ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json
>  PRE-CREATION 
>   
> ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/63793/diff/2/
> 
> 
> Testing
> ---
> 
> done, manually, added new input file for docker env.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63793/
---

(Updated Nov. 14, 2017, 12:50 p.m.)


Review request for Ambari, Krisztian Kasa and Robert Nettleton.


Changes
---

handle that if maxAgeMin value set to zero


Bugs: AMBARI-22395
https://issues.apache.org/jira/browse/AMBARI-22395


Repository: ambari


Description
---

- added new config: "max_age_min" (similar as detach_time_min) to input block. 
that stored in checkpoint files, during cleanup, based on that the checkpoint 
file can be deleted.
- based on "max_age_min" value, during bootstrap, its possible logfeeder wont 
even monitor a file in the future.
- created a new thread which periodically run the checkpoint cleanup (now in 
every 2000 minutes ... its ok to run that rearly)
- fix metrics sink, as its called if some properties are missing
- add new input test files (testing wildcard pattern)
- fix when fatal error thrown if no any files exist based on a wildcard pattern


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
 8548a20 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
 96320e9 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
 8e70850 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java
 3910b9b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java
 322a56d 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java
 cc5d664 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
 2d1bf40 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
 1432c87 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java
 843ae6b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java
 8ee6d00 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a 
  
ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json
 PRE-CREATION 
  
ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63793/diff/2/

Changes: https://reviews.apache.org/r/63793/diff/1-2/


Testing
---

done, manually, added new input file for docker env.


Thanks,

Oliver Szabo

Review Request 63793: Log Feeder: cleanup checkpoint files periodically

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63793/
---

Review request for Ambari, Krisztian Kasa and Robert Nettleton.


Bugs: AMBARI-22395
https://issues.apache.org/jira/browse/AMBARI-22395


Repository: ambari


Description
---

- added new config: "max_age_min" (similar as detach_time_min) to input block. 
that stored in checkpoint files, during cleanup, based on that the checkpoint 
file can be deleted.
- based on "max_age_min" value, during bootstrap, its possible logfeeder wont 
even monitor a file in the future.
- created a new thread which periodically run the checkpoint cleanup (now in 
every 2000 minutes ... its ok to run that rearly)
- fix metrics sink, as its called if some properties are missing
- add new input test files (testing wildcard pattern)
- fix when fatal error thrown if no any files exist based on a wildcard pattern


Diffs
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
 8548a20 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
 96320e9 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
 8e70850 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java
 3910b9b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java
 322a56d 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java
 cc5d664 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
 2d1bf40 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
 1432c87 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java
 843ae6b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java
 8ee6d00 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a 
  
ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json
 PRE-CREATION 
  
ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log
 PRE-CREATION 


Diff: https://reviews.apache.org/r/63793/diff/1/


Testing
---

done, manually, added new input file for docker env.


Thanks,

Oliver Szabo