Davey, Sorry for the delay in getting this done, I was away from the keyboard for almost 1.5 weeks after a death in the family. I have moved the GetTCP processor code to it's own bundle and separate repo [1] with NiFi 1.0.0 as the target. I'm not sure if that is the long term home for it. However, it does feel a little cleaner then mucking with the NiFi Standard Bundle or me making the assumption it would be adopted as part of the standard bundle.
I know of at least one bug in the processor that has been reported -- it fails to reconnect if the connection is severed. I will be working on this, however, I will not be able to get to it till Wednesday night or most likely Thursday morning. Thanks again for your patience and interest. Andrew [1] https://github.com/apsaltis/nifi-gettcp-bundle On Mon, Oct 3, 2016 at 10:56 AM, Davy De Waele <[email protected]> wrote: > Hi Andrew, > > We'd be happy to test and provide feedback. > We have a use-case now with Nifi that we're in the process of implementing. > > If you have sources or binaries feel free to let me know..... > > Regards, > Davy > > On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis <[email protected]> > wrote: > >> Hi Davy, >> Sorry for the slow response, I am traveling today. However, I should be >> able to get a build of it for 1.0.0 out later today or tomorrow. >> >> Thanks, >> Andrew >> >> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <[email protected]> >> wrote: >> >>> Hi Andrew, >>> >>> We tested your 0.0.6 binary and it seemed to do what we want. >>> Unfortunately it is not compatible with the current Nifi version. >>> Do you have a binary available that would work with nifi 1.0.0 ? >>> >>> Thx. >>> >>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis < >>> [email protected]> wrote: >>> >>>> Davy, >>>> The processor I have been working on may meet your needs. You are >>>> correct, at this time I have not pushed the source for it, still working >>>> through some hurdles. The one thing to work out is how you would >>>> dynamically add the processors -- suppose you may be able to use the REST >>>> API for NiFi. I would imagine there are quite a number of these devices >>>> that you would need to have processors for. In the use case I have been >>>> working on, there may be 600 or so endpoints that I need to connect to and >>>> I'm trying to figure out does it make sense to do it this way. >>>> >>>> I'll hopefully be in a place soon that I can push the code I have for >>>> the GetTCP processor. >>>> >>>> >>>> >>>> >>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <[email protected]> wrote: >>>> >>>>> Just wanted to clarify something about ListenTCP... it does support >>>>> multiple incoming connections, however if you using the batch output >>>>> capability, one flow file will contain data across all the connections. >>>>> >>>>> I do agree with Andrew that based on the description it sounds like >>>>> NiFi is expected to be the client that initiates a connection and keeps >>>>> reading for some amount of time/threshold, like a GetTCP processor. >>>>> >>>>> Currently we have ListenTCP which is waiting for incoming connections, >>>>> and PutTCP which makes a connection, but only writes data. >>>>> >>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis < >>>>> [email protected]> wrote: >>>>> >>>>>> Davy, >>>>>> It sounds like you need a GetTCP type of processor that connects from >>>>>> NiFi to the TCP endpoint on the sensor, is that correct? >>>>>> >>>>>> Thanks, >>>>>> Andrew >>>>>> >>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks for the response ... it's an existing network of sensors. The >>>>>>> sensors spit out data over a serial interface that is exposed over a tcp >>>>>>> connection. (rs232 -> ethernet converter in the sensor). >>>>>>> The current sensor architecture involves clients making direct >>>>>>> connections to the individual sensors. (establishing a tcp connection to >>>>>>> the specific ip of the sensor). >>>>>>> >>>>>>> If I understand correctly, ListenTCP would not work in this case for >>>>>>> multiple sensors >>>>>>> >>>>>>> Are you talking about a setup where the sensors would be in a >>>>>>> "client" mode where each sensor would each establish a tcp connections >>>>>>> to a >>>>>>> single ListTCP processor ? >>>>>>> >>>>>>> Thx >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello >>>>>>>> >>>>>>>> Can you talk a bit about why you'd want ListenTCP processors tied >>>>>>>> to a >>>>>>>> given sensor? You should be able to have many sensors to a single >>>>>>>> ListenTCP. Each stream will be between a source/sensor and nifi so >>>>>>>> data won't be getting intermingled there. If we're not providing >>>>>>>> enough session/stream metadata on the flow files to make demux of >>>>>>>> the >>>>>>>> streams easy using something like RouteOnAttribute or whatnot we >>>>>>>> definitely should. >>>>>>>> >>>>>>>> Now, that said, you could certainly programmatically deploy (via the >>>>>>>> REST API) instances of these processors along the lines of what your >>>>>>>> endpoint registry tells you. It just seems on the surface like >>>>>>>> doing >>>>>>>> so would be avoidable at least for the listening of data. Typically >>>>>>>> such a registry would be useful to do additional tagging/enrichment >>>>>>>> of >>>>>>>> the data and would occur once it is in the flow. >>>>>>>> >>>>>>>> Thanks >>>>>>>> Joe >>>>>>>> >>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <[email protected]> >>>>>>>> wrote: >>>>>>>> > We have a large number of sensors that send out data via TCP. The >>>>>>>> idea is to >>>>>>>> > use a ListenTCP processor in Nifi to capture the data, do some >>>>>>>> filtering / >>>>>>>> > basic transformation before sending it upstream into our stack. >>>>>>>> > >>>>>>>> > We can configure individual ListenTCP processors for each sensor, >>>>>>>> and that >>>>>>>> > works fine when the number of sensors is small, but once you hit >>>>>>>> a larger >>>>>>>> > number if becomes cumbersome and difficult to manage. >>>>>>>> > >>>>>>>> > We have an inventory of those sensors (exposed via a REST service >>>>>>>> endpoint), >>>>>>>> > containing the sensor tcp information like ip and port) >>>>>>>> > >>>>>>>> > Is there an easy way to create these ListenTCP processors on the >>>>>>>> fly based >>>>>>>> > on a REST endpoint or some other external configuration ? How >>>>>>>> would that >>>>>>>> > work ? >>>>>>>> > >>>>>>>> > Thx. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Andrew >>>>>> >>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>>> twiiter: @itmdata >>>>>> <http://twitter.com/intent/user?screen_name=itmdata> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Andrew >>>> >>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>>> >>> >>> >> >> >> -- >> Thanks, >> Andrew >> >> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >> > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
