[
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266010#comment-15266010
]
mouwei commented on FLUME-2498:
-------------------------------
Hi,
I fond a bug of this tailsource.
When I use the regular expression to match files under a folder. when some of
file was rolling by log4j, this file's start position which is used to record
tail position will be setted to 0. And then all of matched files will be
readed again.
after checking the code. I find below info:
The process() will update all of inodes info by
"existingInodes.addAll(reader.updateTailFiles());"
But the the skipToEnd will be setted to "false" when update this file.
" public List<Long> updateTailFiles() throws IOException {
return updateTailFiles(false);
}"
when this file was rolled. below code will be executed. this startPos will be
setted to 0. It will be readed again.
if (tf == null || !tf.getPath().equals(f.getAbsolutePath())) {
long startPos = skipToEnd ? f.length() : 0;
tf = openFile(f, headers, inode, startPos);
}
Does anyone occurred same problem or is there any setting I missed?
> Implement Taildir Source
> ------------------------
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch,
> FLUME-2498-4.patch, FLUME-2498-5.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files
> rotate.
> * It periodically writes the last read position of each file in a position
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from
> the position written on the existing position file.
> * It can add event headers to each tailing file group.
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
