[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936321#comment-15936321
]
Hudson commented on AMBARI-20378:
---------------------------------
FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #7102 (See
[https://builds.apache.org/job/Ambari-trunk-Commit/7102/])
AMBARI-20378. Logfeeder: add de-duplication support (oleewere) (oleewere:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=f8931a925f18dccdfe617641ae912fd93cf5f3e1])
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
* (add)
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineFilter.java
* (add)
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/cache/LRUCache.java
* (add)
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/input/cache/LRUCacheTest.java
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperDate.java
* (add)
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputLineFilterTest.java
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/mapper/MapperDateTest.java
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
* (edit) ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties
* (edit)
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-properties.xml
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/filter/FilterJSONTest.java
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputManagerTest.java
* (edit)
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterJSON.java
* (edit)
ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-zookeeper.json
> LogFeeder: Add de-duplication support
> -------------------------------------
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
> Affects Versions: 2.5.0
> Reporter: Olivér Szabó
> Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties (global setting). For example (the
> values below are the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
