[
https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175530#comment-14175530
]
Otis Gospodnetic commented on FLUME-2498:
-----------------------------------------
bq. Yes. They will not be read again.
I can't tell if it's Yes or No to the "will lines be read again" :)
bq. If there is a backlog of many files on start-up, one file will be selected
in random order
Would it be possible to look at the timestamp on unread files? Or see if they
have a numeric extension, like .1, .2, etc. and use some heuristics to try and
read them in the correct order?
bq. No, the consumed files need not be deleted in this source.
I think the person was asking whether this Taildir Source implementation
deletes a file when it's done reading it or not. I think the answer is that it
does NOT delete the file and that file deletion is somebody else's
responsibility. Correct?
> Implement Taildir Source
> ------------------------
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Satoshi Iijima
> Attachments: FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time
> once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files
> rotate.
> * It periodically writes the last read position of each file in a position
> file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from
> the position written on the existing position file.
> * It can add event headers to each tailing file group.
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
