I am currently using NiFi to read in BRO IDS logs using a JSON Streaming plugin. The plugin writes output in json format that rotates its files in 15 min intervals in the format "json_streaming_type.#.log" where the current file is always "json_streaming_typex.log" and the files rotate ascending to the max limit (so 1 becomes 2 and 2 becomes 3 and so on).
Example:
json_streaming_typex.log
json_streaming_typex.1.log
json_streaming_typex.2.log
My TailFile process is configured as follows:
Tailing mode: Multiple Files
File(s) to Tail: json_streaming_[^\.]+\.log
Rolling Filename Pattern: ${filename}.?.log
Base Directory: /opt/logs
Initial Start Position: Beginning of File
State Location: Local
Recursive lookup: false
Lookup Frequency: 10 minuntes
Maximum age: 24 hours
*note: it configured to read every 5 seconds.
This works for a while, but two issues eventually occur. The first is that the state continuously increases and is never cleaned up (I originally assumed this is because the max age is set to 24 hours), and the second is that after an hour or so the state begins to contain entries representing the regular _expression_ used to list the files for tailing. As result of the latter duplication of events occurs.
I am not sure what is going on or why, or what I have done wrong. To be honest I barley understand how the configuration of the TailFile processor is supposed to work when dealing with multiple files that rotate. The documentation is a bit confusing, especially because it makes mention to the "Rolling strategy" attribute that no longer exists.
Today I have reconfigured the setup to change the rolling file name format to "json_streaming_typex.log.#" to align more with provided examples, though it did not change the symptoms/concerns/issues. What has worked, sort-of, is adjusting the Lookup Frequency to 1 minute and the Maximum age to 10 minutes, in addition to reducing the maximum number of rotated files from four to one. The only remaining concern is that the state file continuously grows and I do not know why. Note that the rotation was also adjusted from 15min to 5min to align with the smaller intervals.
Any help in understanding how TailFile manages multiple rolling files and/or understanding why my setup is not working as expected I would greatly appreciate it.
Note: When the state is above 500 entries and one attempts to filter the UI seems to stall.
~Regards
