Hi Andrew, We'd be happy to test and provide feedback. We have a use-case now with Nifi that we're in the process of implementing.
If you have sources or binaries feel free to let me know..... Regards, Davy On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis <[email protected]> wrote: > Hi Davy, > Sorry for the slow response, I am traveling today. However, I should be > able to get a build of it for 1.0.0 out later today or tomorrow. > > Thanks, > Andrew > > On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <[email protected]> wrote: > >> Hi Andrew, >> >> We tested your 0.0.6 binary and it seemed to do what we want. >> Unfortunately it is not compatible with the current Nifi version. >> Do you have a binary available that would work with nifi 1.0.0 ? >> >> Thx. >> >> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis < >> [email protected]> wrote: >> >>> Davy, >>> The processor I have been working on may meet your needs. You are >>> correct, at this time I have not pushed the source for it, still working >>> through some hurdles. The one thing to work out is how you would >>> dynamically add the processors -- suppose you may be able to use the REST >>> API for NiFi. I would imagine there are quite a number of these devices >>> that you would need to have processors for. In the use case I have been >>> working on, there may be 600 or so endpoints that I need to connect to and >>> I'm trying to figure out does it make sense to do it this way. >>> >>> I'll hopefully be in a place soon that I can push the code I have for >>> the GetTCP processor. >>> >>> >>> >>> >>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <[email protected]> wrote: >>> >>>> Just wanted to clarify something about ListenTCP... it does support >>>> multiple incoming connections, however if you using the batch output >>>> capability, one flow file will contain data across all the connections. >>>> >>>> I do agree with Andrew that based on the description it sounds like >>>> NiFi is expected to be the client that initiates a connection and keeps >>>> reading for some amount of time/threshold, like a GetTCP processor. >>>> >>>> Currently we have ListenTCP which is waiting for incoming connections, >>>> and PutTCP which makes a connection, but only writes data. >>>> >>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis < >>>> [email protected]> wrote: >>>> >>>>> Davy, >>>>> It sounds like you need a GetTCP type of processor that connects from >>>>> NiFi to the TCP endpoint on the sensor, is that correct? >>>>> >>>>> Thanks, >>>>> Andrew >>>>> >>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Thanks for the response ... it's an existing network of sensors. The >>>>>> sensors spit out data over a serial interface that is exposed over a tcp >>>>>> connection. (rs232 -> ethernet converter in the sensor). >>>>>> The current sensor architecture involves clients making direct >>>>>> connections to the individual sensors. (establishing a tcp connection to >>>>>> the specific ip of the sensor). >>>>>> >>>>>> If I understand correctly, ListenTCP would not work in this case for >>>>>> multiple sensors >>>>>> >>>>>> Are you talking about a setup where the sensors would be in a >>>>>> "client" mode where each sensor would each establish a tcp connections >>>>>> to a >>>>>> single ListTCP processor ? >>>>>> >>>>>> Thx >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hello >>>>>>> >>>>>>> Can you talk a bit about why you'd want ListenTCP processors tied to >>>>>>> a >>>>>>> given sensor? You should be able to have many sensors to a single >>>>>>> ListenTCP. Each stream will be between a source/sensor and nifi so >>>>>>> data won't be getting intermingled there. If we're not providing >>>>>>> enough session/stream metadata on the flow files to make demux of the >>>>>>> streams easy using something like RouteOnAttribute or whatnot we >>>>>>> definitely should. >>>>>>> >>>>>>> Now, that said, you could certainly programmatically deploy (via the >>>>>>> REST API) instances of these processors along the lines of what your >>>>>>> endpoint registry tells you. It just seems on the surface like doing >>>>>>> so would be avoidable at least for the listening of data. Typically >>>>>>> such a registry would be useful to do additional tagging/enrichment >>>>>>> of >>>>>>> the data and would occur once it is in the flow. >>>>>>> >>>>>>> Thanks >>>>>>> Joe >>>>>>> >>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <[email protected]> >>>>>>> wrote: >>>>>>> > We have a large number of sensors that send out data via TCP. The >>>>>>> idea is to >>>>>>> > use a ListenTCP processor in Nifi to capture the data, do some >>>>>>> filtering / >>>>>>> > basic transformation before sending it upstream into our stack. >>>>>>> > >>>>>>> > We can configure individual ListenTCP processors for each sensor, >>>>>>> and that >>>>>>> > works fine when the number of sensors is small, but once you hit a >>>>>>> larger >>>>>>> > number if becomes cumbersome and difficult to manage. >>>>>>> > >>>>>>> > We have an inventory of those sensors (exposed via a REST service >>>>>>> endpoint), >>>>>>> > containing the sensor tcp information like ip and port) >>>>>>> > >>>>>>> > Is there an easy way to create these ListenTCP processors on the >>>>>>> fly based >>>>>>> > on a REST endpoint or some other external configuration ? How >>>>>>> would that >>>>>>> > work ? >>>>>>> > >>>>>>> > Thx. >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Andrew >>>>> >>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Andrew >>> >>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>> >> >> > > > -- > Thanks, > Andrew > > Subscribe to my book: Streaming Data <http://manning.com/psaltis> > <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> > twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >
