[
https://issues.apache.org/jira/browse/NIFI-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006045#comment-15006045
]
Andre commented on NIFI-1170:
-----------------------------
Mark,
Correct. Just to make it more explicit about the usual use case of people
making use of this:
When you use dates on log files name templates it works more or less like this,
syslog-ng starts writing server2-15-Nov-2015 upon 00:00:00.000 of the 16th, it
opens server2-16-Nov-2015 and starts writing content to it. The old file can
now be compressed and no HUPs are required.
Now comes the more interesting part:
By using a macro like {{$remote-server_$day-$month-$year}} and you can organise
logs by their source, date (and if you want, even by hour!).
This means that while server2-15-Nov-2015 is being written, so is
server3-15-Nov-2015, server4-15-Nov-2015, etc.
Suddenly the syslog admin no longer needs to manually provision syslog sources
on his/her syslog server. Configs get shorter and fewer mistakes occur. Win-Win.
Note:
One could suggest you rotate by the hour and use GetFile but since that
processor needs the files to be closed in order to work properly, this approach
would add latency to the event flow.
> TailFile "File to Tail" property should support Wildcards
> ---------------------------------------------------------
>
> Key: NIFI-1170
> URL: https://issues.apache.org/jira/browse/NIFI-1170
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 0.4.0
> Reporter: Andre
>
> Because of challenges around log rotation of high volume syslog and app
> producers, it is customary to logging platform developers to promote file
> variables based file names such as DynaFiles (rsyslog), Macros(syslog-ng)as
> alternatives to getting SIGHUPs being sent to the syslog daemon upon every
> file rotation.
> (To certain extent, used even NiFi's has similar patterns, like for example,
> when one uses Expression Language to set PutHDFS destination file).
> The current TailFile strategy suggests rotation patterns like:
> {code}
> log_folder/app.log
> log_folder/app.log.1
> log_folder/app.log.2
> log_folder/app.log.3
> {code}
> It is possible to fool the system to accept wildcards by simply using a
> strategy like:
> {code}
> log_folder/test1
> log_folder/server1
> log_folder/server2
> log_folder/server3
> {code}
> And configure *Rolling Filename Pattern* to * but it feels like a hack,
> rather than catering for an ever increasingly prevalent use case
> (DynaFile/macros/etc).
> It would be great if instead, TailFile had the ability to use wildcards on
> File to Tail property
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
