GitHub user tinawenqiao opened a pull request:
https://github.com/apache/flume/pull/54
Support multiline and recursive directory in TaildirSource. And make the
buffersize be configured.
1.Add two parameters in TaildirSourceConfigurationConstants.java:
a. REGEX_START is used for generating Flume events containing multiple
lines in the body, per event. The parameter determines the start of an event.
Default value is "". If the value is set to "", a line with the end of '\n'
will be dealed into one flume event.
b. BUFFER_SIZE is used to define the max number of bytes for one flume
event body's content. Default size is 8192.
2.Put the filePath, hostname, IP into the headers of a flume event if the
headers do not contain the keys.
3.Modify the function getMatchFiles() in ReliableTaildirEventReader.java .
It will support recursive directory.
The sample usage:
agent.sources.taildirsource.filegroups.f1 =
/Users/wenqiao/work/flume/apache-flume-1.7.0-SNAPSHOT-bin/conf/*/01/[ab].log
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tinawenqiao/flume flume-1.7
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flume/pull/54.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #54
----
commit d5ec8c049dba6fb1fef0d15189c5875fadc55e39
Author: wenqiao <315524...@qq.com>
Date: 2016-07-03T02:26:42Z
1.Add two parameters in TaildirSourceConfigurationConstants.java:
a. REGEX_START is used for generating Flume events containing multiple
lines in the body, per event. The parameter determines the start of an event.
Default value is "". If the value is set to "", a line with the end of '\n'
will be dealed into one flume event.
b. BUFFER_SIZE is used to define the max number of bytes for one flume
event body's content. Default size is 8192.
2.Put the filePath, hostname, IP into the headers of a flume event if the
headers do not contain the keys.
3.Modify the function getMatchFiles() in ReliableTaildirEventReader.java .
It will support recursive directory.
The sample usage:
agent.sources.taildirsource.filegroups.f1 =
/Users/wenqiao/work/flume/apache-flume-1.7.0-SNAPSHOT-bin/conf/*/01/[ab].log
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
