Maybe this processor could copy the behavior of the GNU Coreutils "tail" command, aka Linux *tail*? (source is here <http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/tail.c;h=f916d7460395f0cee52c592bc3d160ac94697e73;hb=HEAD> )
If it can, it appears to use the Linux iNotify service to track file changes. If the files are not local or it encounters iNotify problems, it falls back to a manual tracking process. I have never had problems with that utility, and it can track files across logging restarts, rotation, etc. On Thu, Sep 24, 2015 at 7:50 PM, Aldrin Piri (JIRA) <[email protected]> wrote: > > [ > https://issues.apache.org/jira/browse/NIFI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907269#comment-14907269 > ] > > Aldrin Piri commented on NIFI-994: > ---------------------------------- > > I think we can make a best effort at this, but I don't think there are any > guarantees that we have all the data. A lot of this comes down to logging > provider configuration. As an example, perhaps there are constraints on > size or time that cause records to be rotated off. It could take long > outages for these environments to develop, but when the data has been > rolled off, it is gone. Depending on how markers and such work, this > brings up some interesting cases to consider when implementing. > > As another point of consideration, it would be nice to have a > property/properties that provide handling for rolling log formats. > Consider logback and log4j with their date formatted log names. > > > > > Processor to tail files > > ----------------------- > > > > Key: NIFI-994 > > URL: https://issues.apache.org/jira/browse/NIFI-994 > > Project: Apache NiFi > > Issue Type: New Feature > > Affects Versions: 0.4.0 > > Reporter: Joseph Percivall > > > > It's a very common data ingest situation to want to input text into the > system by "tailing" a file, most commonly log files. Currently we don't > have an easy way to do this. > > A simple processor to tail a file would benefit many users. There would > need to be an option to not just tail a file but pick up where the > processor left off if it is interrupted. > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
