[
https://issues.apache.org/jira/browse/FLUME-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163321#comment-15163321
]
Jonathan Smith commented on FLUME-2866:
---------------------------------------
This would also be useful for uploading rolling logs, where we don't want flume
to process the file until the program generating it stops writing to it.
> Add fileTimeMinOffsetSeconds property to Spooling Directory Source
> ------------------------------------------------------------------
>
> Key: FLUME-2866
> URL: https://issues.apache.org/jira/browse/FLUME-2866
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Philippe Lantin
> Priority: Minor
>
> When using a spooling directory source, it would be useful to have the
> ability to specify that files must have a last modified timestamp that is off
> by a configurable amount of seconds, either in the future or the past.
> For example, if I copy a large file to the spooling directory and it takes
> several minutes to copy, I do not want my file to started being processed
> before it is completed. A practical way to do this is by looking at the last
> modified timestamp: files that are being transferred will update this
> timestamp.
> In many filesystems, it is possible for clients to set the time in the
> future, though this is usually done after a file has been completed
> transferred. For example "cp -p" on linux.
> I propose a new property for the Spooling Directory Source:
> fileTimeMinOffsetSeconds. The default would be "0", preserving current
> behavior.
> If fileTimeMinOffsetSeconds=60, files will only be picked up if the last
> modified time is +/- 60 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
