[jira] [Commented] (FLUME-2498) Implement Taildir Source
[ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709844#comment-14709844 ] Haralds Ulmanis commented on FLUME-2498: Ok ... I'm already writing it. Not exactly patch, but another module. Implement Taildir Source Key: FLUME-2498 URL: https://issues.apache.org/jira/browse/FLUME-2498 Project: Flume Issue Type: New Feature Components: Sinks+Sources Reporter: Satoshi Iijima Fix For: v1.7.0 Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498-4.patch, FLUME-2498-5.patch, FLUME-2498.patch This is the proposal of implementing a new tailing source. This source watches the specified files, and tails them in nearly real-time once appends are detected to these files. * This source is reliable and will not miss data even when the tailing files rotate. * It periodically writes the last read position of each file in a position file using the JSON format. * If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file. * It can add event headers to each tailing file group. A attached patch includes a config documentation of this. This source requires Unix-style file system and Java 1.7 or later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2498) Implement Taildir Source
[ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708160#comment-14708160 ] Haralds Ulmanis commented on FLUME-2498: It is supposed to work with regex path .. but if your regex is directory, then it does not work e.g. /var/log/.*/abc.log I did lookup code .. only regex in file name works. Maybe add file manager who will add files to list. e.g. Simplified idea: On start add all directories matching regex (directory part) to inotify ... and then process inotify create events. if dir - add watch if file - add to file list to tail. Regards Implement Taildir Source Key: FLUME-2498 URL: https://issues.apache.org/jira/browse/FLUME-2498 Project: Flume Issue Type: New Feature Components: Sinks+Sources Reporter: Satoshi Iijima Fix For: v1.7.0 Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498-4.patch, FLUME-2498-5.patch, FLUME-2498.patch This is the proposal of implementing a new tailing source. This source watches the specified files, and tails them in nearly real-time once appends are detected to these files. * This source is reliable and will not miss data even when the tailing files rotate. * It periodically writes the last read position of each file in a position file using the JSON format. * If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file. * It can add event headers to each tailing file group. A attached patch includes a config documentation of this. This source requires Unix-style file system and Java 1.7 or later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
