Clay, Multiple packets should not be an issue since it is reading a stream of data from the socket, but I don’t think the prefixed length will work.
The data coming across has to be in a format that one of the record readers can understand. If you have JSON data and then have something additional added to it like the length, then its not going to be readable by a standard JSON library. You could potentially create a custom record reader to handle your format. -Bryan > On Oct 7, 2017, at 10:37 AM, Clay Teahouse <[email protected]> wrote: > > Thank you, Bryan. I was able to set up the flow using your template and > then fix my own. > > Can ListenTCPRecord deal with records spread across multiple packets? More > specifically, can it handle payloads prefixed with the length? I receive > JSON and protobuf messages from a client and each record is prefixed with > the length of the message. > > -Clay > > On Thu, Oct 5, 2017 at 8:22 AM, Bryan Bende <[email protected]> wrote: > >> Have you tried using the template? >> >> https://gist.githubusercontent.com/bbende/fa2bff34e721fef21453986336664c >> b2/raw/db658c64f75fec47785ab63920ee23582bf1492f/multi_line_log_ >> processing.xml >> >> If you import that it will give you the exact flow from my post and >> all you have to do is start everything. >> >> PutTCP is sending to ListenTCP, the connection coming out of PutTCP to >> LogAttribute is for the failure relationship just to easily see if >> anything fails. >> >> -Bryan >> >> >> On Thu, Oct 5, 2017 at 7:05 AM, Clay Teahouse <[email protected]> >> wrote: >>> Thanks Bryan, for the feedback. >>> >>> I don't seem to be able to replicate the example in your blog. So, is >> the >>> flow GenerateFlowFile -> TCPPUT --> LogAttribute and ListenTCPRecord >>> -->LogAttribute? Both TCPPut and ListenTCPRecord are sending data to >>> LogAttribute? Shouldn't ListenTCPRecord be receiving the data data? >> Also, >>> I am not able to link ListenTCPRecord to any other processor. >>> >>> In any case, I tried to send logs to ListenTCPRecord using netcat and >> have >>> ListenTCPRecord send the processed data to LogAttribute, but >>> ListenTCPRecord doesn't seem to be listening (I don't see any activity on >>> the processor). I'd appreciate if you let me what I am doing wrong. >>> >>> thanks >>> Clay >>> >>> On Wed, Oct 4, 2017 at 9:14 AM, Bryan Bende <[email protected]> wrote: >>> >>>> Hello, >>>> >>>> I wrote a post that shows an example of using ListenTCPRecord with a >>>> GrokReader to receive multi-line log messages. There is a link to a >>>> template of the flow at the very end. >>>> >>>> You could easily change the example so that PutTCP is sending a single >>>> JSON document, or an array of JSON documents, and ListenTCPRecord is >>>> using a JsonTreeReader. >>>> >>>> We don't have a protobuf record reader so that currently isn't an >> option. >>>> >>>> Let us know if you have any other questions. >>>> >>>> -Bryan >>>> >>>> [1] https://bryanbende.com/development/2017/10/04/apache- >>>> nifi-processing-multiline-logs >>>> >>>> On Wed, Oct 4, 2017 at 8:39 AM, Clay Teahouse <[email protected]> >>>> wrote: >>>>> Hi All, >>>>> >>>>> Does anyone have an example of ListenTcpRecord processor in action, >> say >>>> for >>>>> example, with a json or protobuf reader? I am specifically wondering >>>> about >>>>> the record length/prefix. >>>>> >>>>> thanks >>>>> Clay >>>> >>
signature.asc
Description: Message signed with OpenPGP
