Daniel Lanza García created FLUME-3219:
------------------------------------------
Summary: Taildir source: if file is rename, it is consumed again
Key: FLUME-3219
URL: https://issues.apache.org/jira/browse/FLUME-3219
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: 1.8.0
Reporter: Daniel Lanza García
Current behavior of Taildir is such that if a file is renamed (eg log rotated)
it is consumed again.
https://github.com/apache/flume/blob/d1f24f56ce9714bb3e1edc671da290c75a17dead/flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java#L247
Would not be better if the inode is followed, and if that inode has been
consumed do not consume it again? With current implementation, once file is
rotated, you get duplicates in the case you include in the path previous days's
data (you want to do that if agent fails and needs to consume data from
previous days).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
