[
https://issues.apache.org/jira/browse/FLUME-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Burgos. Isban (Banco Santander) updated FLUME-2800:
---------------------------------------------------------
Description:
This a proposal of implementation to handle multiline log messages for new
tailing source FLUME-2498.
Based on an idea FLUME-2779 MultiLine Deserializer for Spooling DIrectory
Source.
Config.
* multiLineRegex: Regular expression to handle multiline log messages (grok
expressions permitted)
* grokDictionaryDir: Custom Grok dictionaries directory
* maxNumberLines: Max number of lines per event in multiline log messages.
Default 100. Remaining lines is never transferred to sink.
For Regex expressions use Java regex engine.
Include a functionality for extracting grok expressions into a pure named regex
(inspired by the logstash inteceptor)
By default load the included built-in grok dictionaries with pre-defined
patterns.
https://github.com/aicer/grok
Attached patch includes a config documentation and unit tests.
Also attached a completed port/patch for Flume 1.6 a Java 1.6
was:
This a proposal of implementation to handle multiline log messages for new
tailing source FLUME-2498.
Based on an idea FLUME-2779 MultiLine Deserializer for Spooling DIrectory
Source.
Config.
* multiLineRegex: Regular expression to handle multiline log messages (grok
expressions permitted)
* grokDictionaryDir: Custom Grok dictionaries directory
* maxNumberLines: Max number of lines per event in multiline log messages.
Default 100. Remaining lines is never transferred to sink.
For Regex expressions use joni regex engine which can be twice as fast as the
Java regex engine and will be more efficient, producing less object churn while
scanning, because it operates natively on byte arrays.
https://github.com/jruby/joni
Include a functionality for extracting grok expressions into a pure named regex
(inspired by the logstash inteceptor)
By default load the included built-in grok dictionaries with pre-defined
patterns.
https://github.com/aicer/grok
Attached patch includes a config documentation and unit tests.
Also attached a completed port/patch for Flume 1.6 a Java 1.6
> Multiline log events for Taildir Source
> ---------------------------------------
>
> Key: FLUME-2800
> URL: https://issues.apache.org/jira/browse/FLUME-2800
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.6.0, v1.7.0
> Reporter: David Burgos. Isban (Banco Santander)
> Priority: Minor
> Attachments: FLUME-2800-2-Flume1.6.patch, FLUME-2800-2.patch,
> FLUME-2800-Flume1.6.patch, FLUME-2800.patch
>
>
> This a proposal of implementation to handle multiline log messages for new
> tailing source FLUME-2498.
> Based on an idea FLUME-2779 MultiLine Deserializer for Spooling DIrectory
> Source.
> Config.
> * multiLineRegex: Regular expression to handle multiline log messages (grok
> expressions permitted)
> * grokDictionaryDir: Custom Grok dictionaries directory
> * maxNumberLines: Max number of lines per event in multiline log messages.
> Default 100. Remaining lines is never transferred to sink.
> For Regex expressions use Java regex engine.
> Include a functionality for extracting grok expressions into a pure named
> regex (inspired by the logstash inteceptor)
> By default load the included built-in grok dictionaries with pre-defined
> patterns.
> https://github.com/aicer/grok
> Attached patch includes a config documentation and unit tests.
> Also attached a completed port/patch for Flume 1.6 a Java 1.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
