Bryan - quickly read through the ListenTCP code and associated abstract impl. It did look feasible to enter a series of characters (some string) in some Character set and that it would be able to demarcate the stream. Is there somewhere in there that the hardcoding happens?
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenTCP.java https://github.com/apache/nifi/blob/master/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventBatchingProcessor.java On Wed, Jan 25, 2017 at 5:47 PM, Bryan Bende <[email protected]> wrote: > Raymond, > > Currently ListenTCP uses new line characters to determine logical message > boundaries, and coming out of the processor you can either have 1 logical > message per flow file, or batch together a configurable number of logical > messages into 1 flow file which would be more performant. > > In your case it sounds like you would want to read data until seeing the > "end of data" marker and treat the whole CSV as one logical message. There > is a JIRA to add this capability: https://issues. > apache.org/jira/browse/NIFI-1985 > > I think the best you can do currently is to us a MergeContent processor > somewhere after ListenTCP to merge together the individual lines from the > CSV, but since there is not other information available to tell it how many > total lines there are, it can't guarantee that they are all merged together > in one flow file. You might be able to make some assumptions about the > timing and size of the data and configure MergeContent in such a way that > it should usually get you the whole CSV as one file. > > Hope this helps. > > -Bryan > > On Wed, Jan 25, 2017 at 5:18 PM, Raymond Rogers < > [email protected]> wrote: > >> I'm still new to NiFi and I'm trying to receive text stream containing a >> CSV file of an unknown length (anything from ~100 bytes to almost 300 KB) >> over a TCP socket. The CSV does have an "end of data" marker that I can >> look for but I am unsure of how to accumulate the text until I receive the >> marker and create a flow-file that contains all of the data up to that >> point. >> >> >> >> The data is being sent from an application that cannot changed to use a >> different format. >> >> >> >> Any suggestions? >> >> >> >> >> *Raymond Rogers*Senior Embedded Software Engineer >> >> 15301 N. Dallas Pkwy Suite 500 >> Dallas, TX 75001 >> D: +1 972 744 3928 <(972)%20744-3928> >> rmgnetworks.com <http://www.rmgnetworks.com> >> >> >> >> Notice of Confidentiality: This transmission contains information that >> may be confidential and that may also be privileged. Unless you are the >> intended recipient of the message (or authorized to receive it for the >> intended recipient) you may not copy, forward, or otherwise use it, or >> disclose its contents to anyone else. If you have received this >> transmission in error, please notify us immediately and delete it from your >> system. >> > >
