Davey,
Sorry for the delay in getting this done, I was away from the keyboard for
almost 1.5 weeks after a death in the family. I have moved the GetTCP
processor code to it's own bundle and separate repo [1] with NiFi 1.0.0 as
the target. I'm not sure if that is the long term home for it. However, it
does feel a little cleaner then mucking with the NiFi Standard Bundle or me
making the assumption it would be adopted as part of the standard bundle.

I know of at least one bug in the processor that has been reported -- it
fails to reconnect if the connection is severed. I will be working on this,
however, I will not be able to get to it till Wednesday night or most
likely Thursday morning.

Thanks again for your patience and interest.

Andrew

[1] https://github.com/apsaltis/nifi-gettcp-bundle

On Mon, Oct 3, 2016 at 10:56 AM, Davy De Waele <ddewa...@gmail.com> wrote:

> Hi Andrew,
>
> We'd be happy to test and provide feedback.
> We have a use-case now with Nifi that we're in the process of implementing.
>
> If you have sources or binaries feel free to let me know.....
>
> Regards,
> Davy
>
> On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis <psaltis.and...@gmail.com>
> wrote:
>
>> Hi Davy,
>> Sorry for the slow response, I am traveling today. However, I should be
>> able to get a build of it for 1.0.0 out later today or tomorrow.
>>
>> Thanks,
>> Andrew
>>
>> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <ddewa...@gmail.com>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> We tested your 0.0.6 binary and it seemed to do what we want.
>>> Unfortunately it is not compatible with the current Nifi version.
>>> Do you have a binary available that would work with nifi 1.0.0 ?
>>>
>>> Thx.
>>>
>>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis <
>>> psaltis.and...@gmail.com> wrote:
>>>
>>>> Davy,
>>>> The processor I have been working on may meet your needs. You are
>>>> correct, at this time I have not pushed the source for it, still working
>>>> through some hurdles. The one thing to work out is how you would
>>>> dynamically add the processors -- suppose you may be able to use the REST
>>>> API for NiFi. I would imagine there are quite a number of these devices
>>>> that you would need to have processors for. In the use case I have been
>>>> working on, there may be 600 or so endpoints that I need to connect to and
>>>> I'm trying to figure out does it make sense to do it this way.
>>>>
>>>> I'll hopefully be in a place soon that I can push the code I have for
>>>> the GetTCP processor.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbe...@gmail.com> wrote:
>>>>
>>>>> Just wanted to clarify something about ListenTCP... it does support
>>>>> multiple incoming connections, however if you using the batch output
>>>>> capability, one flow file will contain data across all the connections.
>>>>>
>>>>> I do agree with Andrew that based on the description it sounds like
>>>>> NiFi is expected to be the client that initiates a connection and keeps
>>>>> reading for some amount of time/threshold, like a GetTCP processor.
>>>>>
>>>>> Currently we have ListenTCP which is waiting for incoming connections,
>>>>> and PutTCP which makes a connection, but only writes data.
>>>>>
>>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis <
>>>>> psaltis.and...@gmail.com> wrote:
>>>>>
>>>>>> Davy,
>>>>>> It sounds like you need a GetTCP type of processor that connects from
>>>>>> NiFi to the TCP endpoint on the sensor, is that correct?
>>>>>>
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewa...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks for the response ... it's an existing network of sensors. The
>>>>>>> sensors spit out data over a serial interface that is exposed over a tcp
>>>>>>> connection. (rs232 -> ethernet converter in the sensor).
>>>>>>> The current sensor architecture involves clients making direct
>>>>>>> connections to the individual sensors. (establishing a tcp connection to
>>>>>>> the specific ip of the sensor).
>>>>>>>
>>>>>>> If I understand correctly, ListenTCP would not work in this case for
>>>>>>> multiple sensors
>>>>>>>
>>>>>>> Are you talking about a setup where the sensors would be in a
>>>>>>> "client" mode where each sensor would each establish a tcp connections 
>>>>>>> to a
>>>>>>> single ListTCP processor  ?
>>>>>>>
>>>>>>> Thx
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.w...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>> Can you talk a bit about why you'd want ListenTCP processors tied
>>>>>>>> to a
>>>>>>>> given sensor?  You should be able to have many sensors to a single
>>>>>>>> ListenTCP.  Each stream will be between a source/sensor and nifi so
>>>>>>>> data won't be getting intermingled there.  If we're not providing
>>>>>>>> enough session/stream metadata on the flow files to make demux of
>>>>>>>> the
>>>>>>>> streams easy using something like RouteOnAttribute or whatnot we
>>>>>>>> definitely should.
>>>>>>>>
>>>>>>>> Now, that said, you could certainly programmatically deploy (via the
>>>>>>>> REST API) instances of these processors along the lines of what your
>>>>>>>> endpoint registry tells you.  It just seems on the surface like
>>>>>>>> doing
>>>>>>>> so would be avoidable at least for the listening of data.  Typically
>>>>>>>> such a registry would be useful to do additional tagging/enrichment
>>>>>>>> of
>>>>>>>> the data and would occur once it is in the flow.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Joe
>>>>>>>>
>>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <ddewa...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> > We have a large number of sensors that send out data via TCP. The
>>>>>>>> idea is to
>>>>>>>> > use a ListenTCP processor in Nifi to capture the data, do some
>>>>>>>> filtering /
>>>>>>>> > basic transformation before sending it upstream into our stack.
>>>>>>>> >
>>>>>>>> > We can configure individual ListenTCP processors for each sensor,
>>>>>>>> and that
>>>>>>>> > works fine when the number of sensors is small, but once you hit
>>>>>>>> a larger
>>>>>>>> > number if becomes cumbersome and difficult to manage.
>>>>>>>> >
>>>>>>>> > We have an inventory of those sensors (exposed via a REST service
>>>>>>>> endpoint),
>>>>>>>> > containing  the sensor tcp information like ip and port)
>>>>>>>> >
>>>>>>>> > Is there an easy way to create these ListenTCP processors on the
>>>>>>>> fly based
>>>>>>>> > on a REST endpoint or some other external configuration ? How
>>>>>>>> would that
>>>>>>>> > work ?
>>>>>>>> >
>>>>>>>> > Thx.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Andrew
>>>>>>
>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>>> twiiter: @itmdata
>>>>>> <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Andrew
>>>>
>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks,
>> Andrew
>>
>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>
>
>


-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data <http://manning.com/psaltis>
<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Reply via email to