[
https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426254#comment-13426254
]
Patrick Wendell commented on FLUME-1400:
----------------------------------------
I've looked closely at both the tail C source code and the flume user guide. I
also talked with Mike extensively about this.
The race condition: the tail binary doesn't have this race condition, because
it uses kernel event notification mechanisms to track file movements precisely.
It is the case that a user will need to use "tail -F" to correctly track
rolling logs, but that is consistent with our documentation.
IMHO, a much bigger issue that Mike and I identified is that if a Flume agent
fails or restarts using the execSource(tail), there is near certainty of data
loss. At a minimum I figured this should be made explicit in the User Guide.
I went to add this to the user guide and I found a big red warning box stating
this rather explicitly that was already there.
With that in mind, I am closing as "not an issue". The only documentation
merited by my exploration we already have :)
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail
> -f") when tailing logs, if the log files are themselves being rolled and
> renamed by a writer process.
> The primary concern is whether there is a race condition or correctness
> problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all
> of the ugliness of recognizing when a file is rolled and open the new
> "current" file as expected. However, we should both verify and document that
> behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this
> ugliness by introducing increasing break-fix complexity to the tail source. I
> am hoping that by delegating to the "tail" binary we have avoided having to
> do this (though at the cost of OS portability). Nonetheless, I should confirm
> this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
