Daniel Lanza García updated FLUME-3219:
    Summary: Taildir source: if file is renamed, it is consumed again  (was: 
Taildir source: if file is rename, it is consumed again)

> Taildir source: if file is renamed, it is consumed again
> --------------------------------------------------------
>                 Key: FLUME-3219
>                 URL: https://issues.apache.org/jira/browse/FLUME-3219
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.8.0
>            Reporter: Daniel Lanza García
>            Priority: Major
> Current behavior of Taildir is such that if a file is renamed (eg log 
> rotated) it is consumed again.
> https://github.com/apache/flume/blob/d1f24f56ce9714bb3e1edc671da290c75a17dead/flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java#L247
> Would not be better if the inode is followed, and if that inode has been 
> consumed do not consume it again? With current implementation, once file is 
> rotated, you get duplicates in the case you include in the path previous 
> days's data (you want to do that if agent fails and needs to consume data 
> from previous days).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org
For additional commands, e-mail: issues-h...@flume.apache.org

Reply via email to