Hey Matthew, Sorry about the confusion. I think the problem is around the "Tailing mode" property. In your case, you want to use "Single file" because your desire is to tail a single "rolling" file. You'd want to use "Multiple files" if you were tailing multiple independent files.
So you'll want to set "Tailing Mode" to "Single File" and then set the "File to Tail" to "json_streaming_typex.log" and the "Rolling Filename Pattern" to "json_streaming_typex\.\d+\.log" if I am understanding the naming pattern correctly. I hope this helps to clarify. If not, please let us know. Thanks -Mark On Sep 19, 2018, at 10:21 AM, Matthew Gaetano <[email protected]<mailto:[email protected]>> wrote: I am currently using NiFi to read in BRO IDS logs using a JSON Streaming plugin. The plugin writes output in json format that rotates its files in 15 min intervals in the format "json_streaming_type.#.log" where the current file is always "json_streaming_typex.log" and the files rotate ascending to the max limit (so 1 becomes 2 and 2 becomes 3 and so on). Example: json_streaming_typex.log json_streaming_typex.1.log json_streaming_typex.2.log My TailFile process is configured as follows: Tailing mode: Multiple Files File(s) to Tail: json_streaming_[^\.]+\.log Rolling Filename Pattern: ${filename}.?.log Base Directory: /opt/logs Initial Start Position: Beginning of File State Location: Local Recursive lookup: false Lookup Frequency: 10 minuntes Maximum age: 24 hours *note: it configured to read every 5 seconds. This works for a while, but two issues eventually occur. The first is that the state continuously increases and is never cleaned up (I originally assumed this is because the max age is set to 24 hours), and the second is that after an hour or so the state begins to contain entries representing the regular expression used to list the files for tailing. As result of the latter duplication of events occurs. I am not sure what is going on or why, or what I have done wrong. To be honest I barley understand how the configuration of the TailFile processor is supposed to work when dealing with multiple files that rotate. The documentation is a bit confusing, especially because it makes mention to the "Rolling strategy" attribute that no longer exists. Today I have reconfigured the setup to change the rolling file name format to "json_streaming_typex.log.#" to align more with provided examples, though it did not change the symptoms/concerns/issues. What has worked, sort-of, is adjusting the Lookup Frequency to 1 minute and the Maximum age to 10 minutes, in addition to reducing the maximum number of rotated files from four to one. The only remaining concern is that the state file continuously grows and I do not know why. Note that the rotation was also adjusted from 15min to 5min to align with the smaller intervals. Any help in understanding how TailFile manages multiple rolling files and/or understanding why my setup is not working as expected I would greatly appreciate it. Note: When the state is above 500 entries and one attempts to filter the UI seems to stall. ~Regards
