Daniel Lanza García created FLUME-3219:
------------------------------------------

             Summary: Taildir source: if file is rename, it is consumed again
                 Key: FLUME-3219
                 URL: https://issues.apache.org/jira/browse/FLUME-3219
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: 1.8.0
            Reporter: Daniel Lanza García


Current behavior of Taildir is such that if a file is renamed (eg log rotated) 
it is consumed again.

https://github.com/apache/flume/blob/d1f24f56ce9714bb3e1edc671da290c75a17dead/flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java#L247

Would not be better if the inode is followed, and if that inode has been 
consumed do not consume it again? With current implementation, once file is 
rotated, you get duplicates in the case you include in the path previous days's 
data (you want to do that if agent fails and needs to consume data from 
previous days).

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org
For additional commands, e-mail: issues-h...@flume.apache.org

Reply via email to