[
https://issues.apache.org/jira/browse/NIFI-8773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-8773:
-----------------------------
Summary: Allow TailFile to hold off on ingesting lines of text if the full
(multi-line) message is not available (was: Allow TailFile to hold off on
ingesting lines of text if the fully message is not available)
> Allow TailFile to hold off on ingesting lines of text if the full
> (multi-line) message is not available
> -------------------------------------------------------------------------------------------------------
>
> Key: NIFI-8773
> URL: https://issues.apache.org/jira/browse/NIFI-8773
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 1.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When using TailFile, there are times when multi-line messages are written to
> a file. For example, we may have something like:
> {code}
> <1> My Message
> <2> My Message
> <3> My Message
> A continuation of my message
> {code}
> If TailFile now runs, it will ingest these 4 lines of text as a FlowFile.
> Perhaps the next lines to get written, though, will be something like:
> {code}
> Another continuation of my message
> A final continuation
> <4> Another Message
> <5> Yet another Message
> {code}
> And we may want to avoid pulling in lines "<3> My Message" and " A
> continuation of my message" until we are able to fully consume the full
> message.
> We should enable this capability by allowing for a new property that
> specifies a Regular Expression to run against the start of a line. If we read
> a line from the file and it matches that Regex, then we know the previous
> message is complete. Otherwise, the previous message may not be complete and
> should be buffered (up to some configurable limit, in order to avoid
> exhausting the Java heap).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
