Hi Davy, Thanks for the P/R - I merged that in just a little bit ago. I think the feature you are describing makes total sense, I'm sure this would be useful in many different use cases. I am in the process of wrapping up the changes to fulfill JIRA NIFI-2615 [1]. Do you want to add the "delimiter based spliter" feature to that JIRA? Are you planning on implementing it or are you asking if I would?
I should have the updated bits up in GH this evening/very early am. Thanks your kinda words regarding our loss. Thanks, Andrew On Sun, Oct 30, 2016 at 7:26 PM, Davy De Waele <[email protected]> wrote: > Hi Andrew, > > I've looked at your code and submitted a pull request as I needed to do > some code fixes to get it working > Also posted an issue / question on how you see this GetTCP processor > evolve. > > We're using Nifi to capture sensor data over TCP so a good GetTCP is > definitely something we need in our Nifi tool-belt > > We would like to use it in a slightly different manner as per the current > implementation. We would like to be able to specify a delimiter (byte > sequence) in order to generate flow files out of this processor. > Our sensor devices use a delimiter (end of transmission byte) to mark the > end of a message in the tcp stream. > > That way, the GetTCP processor would already "split" the tcp stream into > separate FlowFiles where each FlowFile would correspond to a response > received from the tcp stream. I've seen something similar in the ListenTCP > processor. > > Sorry to hear about your loss. My condolences. > > Regards, > Davy > > > > On Tue, Oct 18, 2016 at 10:16 AM, Andrew Psaltis <[email protected] > > wrote: > >> Davey, >> Sorry for the delay in getting this done, I was away from the keyboard >> for almost 1.5 weeks after a death in the family. I have moved the GetTCP >> processor code to it's own bundle and separate repo [1] with NiFi 1.0.0 as >> the target. I'm not sure if that is the long term home for it. However, it >> does feel a little cleaner then mucking with the NiFi Standard Bundle or me >> making the assumption it would be adopted as part of the standard bundle. >> >> I know of at least one bug in the processor that has been reported -- it >> fails to reconnect if the connection is severed. I will be working on this, >> however, I will not be able to get to it till Wednesday night or most >> likely Thursday morning. >> >> Thanks again for your patience and interest. >> >> Andrew >> >> [1] https://github.com/apsaltis/nifi-gettcp-bundle >> >> On Mon, Oct 3, 2016 at 10:56 AM, Davy De Waele <[email protected]> >> wrote: >> >>> Hi Andrew, >>> >>> We'd be happy to test and provide feedback. >>> We have a use-case now with Nifi that we're in the process of >>> implementing. >>> >>> If you have sources or binaries feel free to let me know..... >>> >>> Regards, >>> Davy >>> >>> On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis < >>> [email protected]> wrote: >>> >>>> Hi Davy, >>>> Sorry for the slow response, I am traveling today. However, I should be >>>> able to get a build of it for 1.0.0 out later today or tomorrow. >>>> >>>> Thanks, >>>> Andrew >>>> >>>> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <[email protected]> >>>> wrote: >>>> >>>>> Hi Andrew, >>>>> >>>>> We tested your 0.0.6 binary and it seemed to do what we want. >>>>> Unfortunately it is not compatible with the current Nifi version. >>>>> Do you have a binary available that would work with nifi 1.0.0 ? >>>>> >>>>> Thx. >>>>> >>>>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis < >>>>> [email protected]> wrote: >>>>> >>>>>> Davy, >>>>>> The processor I have been working on may meet your needs. You are >>>>>> correct, at this time I have not pushed the source for it, still working >>>>>> through some hurdles. The one thing to work out is how you would >>>>>> dynamically add the processors -- suppose you may be able to use the REST >>>>>> API for NiFi. I would imagine there are quite a number of these devices >>>>>> that you would need to have processors for. In the use case I have been >>>>>> working on, there may be 600 or so endpoints that I need to connect to >>>>>> and >>>>>> I'm trying to figure out does it make sense to do it this way. >>>>>> >>>>>> I'll hopefully be in a place soon that I can push the code I have for >>>>>> the GetTCP processor. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Just wanted to clarify something about ListenTCP... it does support >>>>>>> multiple incoming connections, however if you using the batch output >>>>>>> capability, one flow file will contain data across all the connections. >>>>>>> >>>>>>> I do agree with Andrew that based on the description it sounds like >>>>>>> NiFi is expected to be the client that initiates a connection and keeps >>>>>>> reading for some amount of time/threshold, like a GetTCP processor. >>>>>>> >>>>>>> Currently we have ListenTCP which is waiting for incoming >>>>>>> connections, and PutTCP which makes a connection, but only writes data. >>>>>>> >>>>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Davy, >>>>>>>> It sounds like you need a GetTCP type of processor that connects >>>>>>>> from NiFi to the TCP endpoint on the sensor, is that correct? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Andrew >>>>>>>> >>>>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Thanks for the response ... it's an existing network of sensors. >>>>>>>>> The sensors spit out data over a serial interface that is exposed >>>>>>>>> over a >>>>>>>>> tcp connection. (rs232 -> ethernet converter in the sensor). >>>>>>>>> The current sensor architecture involves clients making direct >>>>>>>>> connections to the individual sensors. (establishing a tcp connection >>>>>>>>> to >>>>>>>>> the specific ip of the sensor). >>>>>>>>> >>>>>>>>> If I understand correctly, ListenTCP would not work in this case >>>>>>>>> for multiple sensors >>>>>>>>> >>>>>>>>> Are you talking about a setup where the sensors would be in a >>>>>>>>> "client" mode where each sensor would each establish a tcp >>>>>>>>> connections to a >>>>>>>>> single ListTCP processor ? >>>>>>>>> >>>>>>>>> Thx >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello >>>>>>>>>> >>>>>>>>>> Can you talk a bit about why you'd want ListenTCP processors tied >>>>>>>>>> to a >>>>>>>>>> given sensor? You should be able to have many sensors to a single >>>>>>>>>> ListenTCP. Each stream will be between a source/sensor and nifi >>>>>>>>>> so >>>>>>>>>> data won't be getting intermingled there. If we're not providing >>>>>>>>>> enough session/stream metadata on the flow files to make demux of >>>>>>>>>> the >>>>>>>>>> streams easy using something like RouteOnAttribute or whatnot we >>>>>>>>>> definitely should. >>>>>>>>>> >>>>>>>>>> Now, that said, you could certainly programmatically deploy (via >>>>>>>>>> the >>>>>>>>>> REST API) instances of these processors along the lines of what >>>>>>>>>> your >>>>>>>>>> endpoint registry tells you. It just seems on the surface like >>>>>>>>>> doing >>>>>>>>>> so would be avoidable at least for the listening of data. >>>>>>>>>> Typically >>>>>>>>>> such a registry would be useful to do additional >>>>>>>>>> tagging/enrichment of >>>>>>>>>> the data and would occur once it is in the flow. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Joe >>>>>>>>>> >>>>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> > We have a large number of sensors that send out data via TCP. >>>>>>>>>> The idea is to >>>>>>>>>> > use a ListenTCP processor in Nifi to capture the data, do some >>>>>>>>>> filtering / >>>>>>>>>> > basic transformation before sending it upstream into our stack. >>>>>>>>>> > >>>>>>>>>> > We can configure individual ListenTCP processors for each >>>>>>>>>> sensor, and that >>>>>>>>>> > works fine when the number of sensors is small, but once you >>>>>>>>>> hit a larger >>>>>>>>>> > number if becomes cumbersome and difficult to manage. >>>>>>>>>> > >>>>>>>>>> > We have an inventory of those sensors (exposed via a REST >>>>>>>>>> service endpoint), >>>>>>>>>> > containing the sensor tcp information like ip and port) >>>>>>>>>> > >>>>>>>>>> > Is there an easy way to create these ListenTCP processors on >>>>>>>>>> the fly based >>>>>>>>>> > on a REST endpoint or some other external configuration ? How >>>>>>>>>> would that >>>>>>>>>> > work ? >>>>>>>>>> > >>>>>>>>>> > Thx. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks, >>>>>>>> Andrew >>>>>>>> >>>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>>>>> twiiter: @itmdata >>>>>>>> <http://twitter.com/intent/user?screen_name=itmdata> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Andrew >>>>>> >>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>>> twiiter: @itmdata >>>>>> <http://twitter.com/intent/user?screen_name=itmdata> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Andrew >>>> >>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>>> >>> >>> >> >> >> -- >> Thanks, >> Andrew >> >> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >> > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
