Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

This is a shot in the dark, but I'd be curious if you saw any
different behavior running your test with this change:

https://github.com/apache/nifi/pull/2168



On Thu, Sep 21, 2017 at 1:31 AM, ddewaele  wrote:
> After looking at the tcp dumps (data packets) the only thing I noticed is
> that the delimiter is now sent in a separate dedicated tcp frame. The rest
> goes beyond my tcp knowledge level :)
>
> I can send you the dumps on your gmail if you want ?
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Davy De Waele]

The issue is on a CentOS linux server, running Nifi 1.4.0 snapshot. There
it communicates with a dozen of Moxa nports over tcp on various remote
sites. All moxa's are connected over a (slow) cellular connection.

Will do a tcpdump with the new / old PutTCP processor.

On Wed, 20 Sep 2017 at 22:10, Bryan Bende  wrote:

> The sender/output stream can only ever be in use by one thread since
> it acquires it from the pool at the beginning of onTrigger, which
> makes it unavailable to any other threads until it is placed back in
> the pool.
>
> The flush is there just because now it is writing to an output stream
> and we may be leaving the sender/output stream open depending if
> "connection per flow file" is true or not.
>
> In the case where you do produce the issue, where is NiFi running? Is
> it sending from NiFi on your Mac to the device on Linux?
>
> On Wed, Sep 20, 2017 at 3:45 PM, Davy De Waele  wrote:
> > Don't get it either... new code looks more or less the same. been trying
> > various things all day. Glad I got it working with a "patched" PutTCP.
> > Better from what I got but still not ideal.
> >
> > Can it be a threading issue (we interact with multiple modules) in the
> input
> > / output stream handling ? Also noticed a flush that wasn't there before.
> > Are there other places where charsets can be set ?
> >
> > Been thinking about what could cause the tcp server to react this way
> but no
> > idea. Can reproduce it easily. Will check if I can enable additional
> logging
> > on the moxa.
> >
> > On Wed, 20 Sep 2017 at 21:36, Bryan Bende  wrote:
> >>
> >> For what it is worth, I can't reproduce this on my Mac laptop either...
> >>
> >> Setup GenerateFlowFile with same sample text your provided, followed
> >> by PutTCP with the \r\n delimiter, this sends to ListenTCP in the same
> >> NiFi instance.
> >>
> >> If I look at the content of the flow files coming out of ListenTCP,
> >> and I put the content viewer in Hex mode, I see:
> >>
> >> 0x 0D 40 43 4F 4D 4D 41 4E 44 0D 0D .@COMMAND..
> >>
> >> The 0D 40 43 4F 4D 4D 41 4E 44 0D 0D lines up with \r@COMMAND\r  which
> >> would make sense since ListenTCP read-off the \r\n at the end.
> >>
> >>
> >> On Wed, Sep 20, 2017 at 1:20 PM, Bryan Bende  wrote:
> >> > I'll keep looking at this, but just to summarize the changes that were
> >> > made in case something jumps out to someone else...
> >> >
> >> > The changes were to avoid reading the entire content of the flow file
> >> > into memory.
> >> >
> >> > Previous Behavior:
> >> > - Create a ByteArrayOuputStream
> >> > - Use an InputStreamCallback to read entire bytes of flow file, and
> >> > write them to above output stream
> >> > - Get value of the outgoing delimiter property
> >> > - If delimiter is not null, then get the bytes of the delimiter using
> >> > the charset configured in the processor, and write the delimiter bytes
> >> > to the above output stream
> >> > - Get byte array from the ByteArrayOutputStream and pass it to the
> >> > channel sender, which in turns calls write on a SocketChannel
> >> >
> >> > New behavior:
> >> > - Get the direct SockeOutputStream from the sender
> >> > - Get the InputStream of the flow file
> >> > - Call IOUtils.copy to copy the flow file input stream to the socket
> >> > output stream in chunks
> >> > - Perform exact same delimiter logic as above
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Sep 20, 2017 at 1:13 PM, Davy De Waele 
> >> > wrote:
> >> >> Correct about the delimiter.
> >> >> Think character set is (was) automatically filled in with utf-8.
> (Never
> >> >> noticed a default option)
> >> >>
> >> >> On Wed, 20 Sep 2017 at 19:11, Bryan Bende  wrote:
> >> >>>
> >> >>> Thanks for the info and for digging into this..
> >> >>>
> >> >>> Just to confirm, in your PutTCP configuration, you are saying that
> you
> >> >>> have "Outgoing Message Delimiter" set to \r\n  ?
> >> >>>
> >> >>> And do you have the "Character Set" property set to the default of
> >> >>> "UTF-8"
> >> >>> ?
> >> >>>
> >> >>> On Wed, Sep 20, 2017 at 12:59 PM, ddewaele 
> wrote:
> >> >>> > I've put up a git repo that I use to test the PutTCP behavior.
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > (
> https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1
> )
> >> >>> >
> >> >>> > What I've done.
> >> >>> >
> >> >>> > - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my base
> nifi
> >> >>> > installation
> >> >>> > - Noticed weird characters in the tcp responses
> >> >>> > - Switched to 1.3.0. No more weird characters
> >> >>> > - Isolated the PutTCP processor by creating a custom PutTCP (see
> >> >>> > github
> >> >>> > repo
> >> >>> > below) to experiment (instead of the built-in PutTCP)
> >> >>> > - Repo has 2 branches. master contains an "older" PutTCP processor
> >> >>> > from
> >> >>> > the
> >> >>> > Nifi code base. 

Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

The sender/output stream can only ever be in use by one thread since
it acquires it from the pool at the beginning of onTrigger, which
makes it unavailable to any other threads until it is placed back in
the pool.

The flush is there just because now it is writing to an output stream
and we may be leaving the sender/output stream open depending if
"connection per flow file" is true or not.

In the case where you do produce the issue, where is NiFi running? Is
it sending from NiFi on your Mac to the device on Linux?

On Wed, Sep 20, 2017 at 3:45 PM, Davy De Waele  wrote:
> Don't get it either... new code looks more or less the same. been trying
> various things all day. Glad I got it working with a "patched" PutTCP.
> Better from what I got but still not ideal.
>
> Can it be a threading issue (we interact with multiple modules) in the input
> / output stream handling ? Also noticed a flush that wasn't there before.
> Are there other places where charsets can be set ?
>
> Been thinking about what could cause the tcp server to react this way but no
> idea. Can reproduce it easily. Will check if I can enable additional logging
> on the moxa.
>
> On Wed, 20 Sep 2017 at 21:36, Bryan Bende  wrote:
>>
>> For what it is worth, I can't reproduce this on my Mac laptop either...
>>
>> Setup GenerateFlowFile with same sample text your provided, followed
>> by PutTCP with the \r\n delimiter, this sends to ListenTCP in the same
>> NiFi instance.
>>
>> If I look at the content of the flow files coming out of ListenTCP,
>> and I put the content viewer in Hex mode, I see:
>>
>> 0x 0D 40 43 4F 4D 4D 41 4E 44 0D 0D .@COMMAND..
>>
>> The 0D 40 43 4F 4D 4D 41 4E 44 0D 0D lines up with \r@COMMAND\r  which
>> would make sense since ListenTCP read-off the \r\n at the end.
>>
>>
>> On Wed, Sep 20, 2017 at 1:20 PM, Bryan Bende  wrote:
>> > I'll keep looking at this, but just to summarize the changes that were
>> > made in case something jumps out to someone else...
>> >
>> > The changes were to avoid reading the entire content of the flow file
>> > into memory.
>> >
>> > Previous Behavior:
>> > - Create a ByteArrayOuputStream
>> > - Use an InputStreamCallback to read entire bytes of flow file, and
>> > write them to above output stream
>> > - Get value of the outgoing delimiter property
>> > - If delimiter is not null, then get the bytes of the delimiter using
>> > the charset configured in the processor, and write the delimiter bytes
>> > to the above output stream
>> > - Get byte array from the ByteArrayOutputStream and pass it to the
>> > channel sender, which in turns calls write on a SocketChannel
>> >
>> > New behavior:
>> > - Get the direct SockeOutputStream from the sender
>> > - Get the InputStream of the flow file
>> > - Call IOUtils.copy to copy the flow file input stream to the socket
>> > output stream in chunks
>> > - Perform exact same delimiter logic as above
>> >
>> >
>> >
>> >
>> > On Wed, Sep 20, 2017 at 1:13 PM, Davy De Waele 
>> > wrote:
>> >> Correct about the delimiter.
>> >> Think character set is (was) automatically filled in with utf-8. (Never
>> >> noticed a default option)
>> >>
>> >> On Wed, 20 Sep 2017 at 19:11, Bryan Bende  wrote:
>> >>>
>> >>> Thanks for the info and for digging into this..
>> >>>
>> >>> Just to confirm, in your PutTCP configuration, you are saying that you
>> >>> have "Outgoing Message Delimiter" set to \r\n  ?
>> >>>
>> >>> And do you have the "Character Set" property set to the default of
>> >>> "UTF-8"
>> >>> ?
>> >>>
>> >>> On Wed, Sep 20, 2017 at 12:59 PM, ddewaele  wrote:
>> >>> > I've put up a git repo that I use to test the PutTCP behavior.
>> >>> >
>> >>> >
>> >>> >
>> >>> > (https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1)
>> >>> >
>> >>> > What I've done.
>> >>> >
>> >>> > - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my base nifi
>> >>> > installation
>> >>> > - Noticed weird characters in the tcp responses
>> >>> > - Switched to 1.3.0. No more weird characters
>> >>> > - Isolated the PutTCP processor by creating a custom PutTCP (see
>> >>> > github
>> >>> > repo
>> >>> > below) to experiment (instead of the built-in PutTCP)
>> >>> > - Repo has 2 branches. master contains an "older" PutTCP processor
>> >>> > from
>> >>> > the
>> >>> > Nifi code base. With that one I don't have the weird characters
>> >>> > (even on
>> >>> > a
>> >>> > nifi 1.4.0-SNAPSHOT + your PR).
>> >>> > - If I use the dev_puttcp_1.4 branch on my nifi I get the weird
>> >>> > characters.
>> >>> >
>> >>> >
>> >>> > Some info on my setup :
>> >>> >
>> >>> > I'm sending commands to PutTCP using a GenerateFlowFile processor.
>> >>> > payload = ${literal('\r')}@COMMAND${literal('\r')}
>> >>> > PutTCP adds a delimier (\r\n)
>> >>> >
>> >>> > Regarding provenance. PutTCP adds the delimiter to the payload
>> >>> > before
>> >>> > sending it to the TCP 

Re: PutTCP connector not cleaning up dangling connections

[Davy De Waele]

Don't get it either... new code looks more or less the same. been trying
various things all day. Glad I got it working with a "patched" PutTCP.
Better from what I got but still not ideal.

Can it be a threading issue (we interact with multiple modules) in the
input / output stream handling ? Also noticed a flush that wasn't there
before. Are there other places where charsets can be set ?

Been thinking about what could cause the tcp server to react this way but
no idea. Can reproduce it easily. Will check if I can enable additional
logging on the moxa.

On Wed, 20 Sep 2017 at 21:36, Bryan Bende  wrote:

> For what it is worth, I can't reproduce this on my Mac laptop either...
>
> Setup GenerateFlowFile with same sample text your provided, followed
> by PutTCP with the \r\n delimiter, this sends to ListenTCP in the same
> NiFi instance.
>
> If I look at the content of the flow files coming out of ListenTCP,
> and I put the content viewer in Hex mode, I see:
>
> 0x 0D 40 43 4F 4D 4D 41 4E 44 0D 0D .@COMMAND..
>
> The 0D 40 43 4F 4D 4D 41 4E 44 0D 0D lines up with \r@COMMAND\r  which
> would make sense since ListenTCP read-off the \r\n at the end.
>
>
> On Wed, Sep 20, 2017 at 1:20 PM, Bryan Bende  wrote:
> > I'll keep looking at this, but just to summarize the changes that were
> > made in case something jumps out to someone else...
> >
> > The changes were to avoid reading the entire content of the flow file
> > into memory.
> >
> > Previous Behavior:
> > - Create a ByteArrayOuputStream
> > - Use an InputStreamCallback to read entire bytes of flow file, and
> > write them to above output stream
> > - Get value of the outgoing delimiter property
> > - If delimiter is not null, then get the bytes of the delimiter using
> > the charset configured in the processor, and write the delimiter bytes
> > to the above output stream
> > - Get byte array from the ByteArrayOutputStream and pass it to the
> > channel sender, which in turns calls write on a SocketChannel
> >
> > New behavior:
> > - Get the direct SockeOutputStream from the sender
> > - Get the InputStream of the flow file
> > - Call IOUtils.copy to copy the flow file input stream to the socket
> > output stream in chunks
> > - Perform exact same delimiter logic as above
> >
> >
> >
> >
> > On Wed, Sep 20, 2017 at 1:13 PM, Davy De Waele 
> wrote:
> >> Correct about the delimiter.
> >> Think character set is (was) automatically filled in with utf-8. (Never
> >> noticed a default option)
> >>
> >> On Wed, 20 Sep 2017 at 19:11, Bryan Bende  wrote:
> >>>
> >>> Thanks for the info and for digging into this..
> >>>
> >>> Just to confirm, in your PutTCP configuration, you are saying that you
> >>> have "Outgoing Message Delimiter" set to \r\n  ?
> >>>
> >>> And do you have the "Character Set" property set to the default of
> "UTF-8"
> >>> ?
> >>>
> >>> On Wed, Sep 20, 2017 at 12:59 PM, ddewaele  wrote:
> >>> > I've put up a git repo that I use to test the PutTCP behavior.
> >>> >
> >>> >
> >>> > (
> https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1
> )
> >>> >
> >>> > What I've done.
> >>> >
> >>> > - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my base nifi
> >>> > installation
> >>> > - Noticed weird characters in the tcp responses
> >>> > - Switched to 1.3.0. No more weird characters
> >>> > - Isolated the PutTCP processor by creating a custom PutTCP (see
> github
> >>> > repo
> >>> > below) to experiment (instead of the built-in PutTCP)
> >>> > - Repo has 2 branches. master contains an "older" PutTCP processor
> from
> >>> > the
> >>> > Nifi code base. With that one I don't have the weird characters
> (even on
> >>> > a
> >>> > nifi 1.4.0-SNAPSHOT + your PR).
> >>> > - If I use the dev_puttcp_1.4 branch on my nifi I get the weird
> >>> > characters.
> >>> >
> >>> >
> >>> > Some info on my setup :
> >>> >
> >>> > I'm sending commands to PutTCP using a GenerateFlowFile processor.
> >>> > payload = ${literal('\r')}@COMMAND${literal('\r')}
> >>> > PutTCP adds a delimier (\r\n)
> >>> >
> >>> > Regarding provenance. PutTCP adds the delimiter to the payload before
> >>> > sending it to the TCP socket. That  delimiter is not in the flowfile
> >>> > that is
> >>> > put in provenance I think.
> >>> >
> >>> > Cannot reproduce the issue on my mac with my mock tcp server. But on
> the
> >>> > actual environment (Linux CentOS) with the Moxa devices, there is a
> >>> > clear
> >>> > difference in behaviour. Will see if I can tunnel my through the
> devices
> >>> >
> >>> > All the info I have right now 
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

For what it is worth, I can't reproduce this on my Mac laptop either...

Setup GenerateFlowFile with same sample text your provided, followed
by PutTCP with the \r\n delimiter, this sends to ListenTCP in the same
NiFi instance.

If I look at the content of the flow files coming out of ListenTCP,
and I put the content viewer in Hex mode, I see:

0x 0D 40 43 4F 4D 4D 41 4E 44 0D 0D .@COMMAND..

The 0D 40 43 4F 4D 4D 41 4E 44 0D 0D lines up with \r@COMMAND\r  which
would make sense since ListenTCP read-off the \r\n at the end.


On Wed, Sep 20, 2017 at 1:20 PM, Bryan Bende  wrote:
> I'll keep looking at this, but just to summarize the changes that were
> made in case something jumps out to someone else...
>
> The changes were to avoid reading the entire content of the flow file
> into memory.
>
> Previous Behavior:
> - Create a ByteArrayOuputStream
> - Use an InputStreamCallback to read entire bytes of flow file, and
> write them to above output stream
> - Get value of the outgoing delimiter property
> - If delimiter is not null, then get the bytes of the delimiter using
> the charset configured in the processor, and write the delimiter bytes
> to the above output stream
> - Get byte array from the ByteArrayOutputStream and pass it to the
> channel sender, which in turns calls write on a SocketChannel
>
> New behavior:
> - Get the direct SockeOutputStream from the sender
> - Get the InputStream of the flow file
> - Call IOUtils.copy to copy the flow file input stream to the socket
> output stream in chunks
> - Perform exact same delimiter logic as above
>
>
>
>
> On Wed, Sep 20, 2017 at 1:13 PM, Davy De Waele  wrote:
>> Correct about the delimiter.
>> Think character set is (was) automatically filled in with utf-8. (Never
>> noticed a default option)
>>
>> On Wed, 20 Sep 2017 at 19:11, Bryan Bende  wrote:
>>>
>>> Thanks for the info and for digging into this..
>>>
>>> Just to confirm, in your PutTCP configuration, you are saying that you
>>> have "Outgoing Message Delimiter" set to \r\n  ?
>>>
>>> And do you have the "Character Set" property set to the default of "UTF-8"
>>> ?
>>>
>>> On Wed, Sep 20, 2017 at 12:59 PM, ddewaele  wrote:
>>> > I've put up a git repo that I use to test the PutTCP behavior.
>>> >
>>> >
>>> > (https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1)
>>> >
>>> > What I've done.
>>> >
>>> > - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my base nifi
>>> > installation
>>> > - Noticed weird characters in the tcp responses
>>> > - Switched to 1.3.0. No more weird characters
>>> > - Isolated the PutTCP processor by creating a custom PutTCP (see github
>>> > repo
>>> > below) to experiment (instead of the built-in PutTCP)
>>> > - Repo has 2 branches. master contains an "older" PutTCP processor from
>>> > the
>>> > Nifi code base. With that one I don't have the weird characters (even on
>>> > a
>>> > nifi 1.4.0-SNAPSHOT + your PR).
>>> > - If I use the dev_puttcp_1.4 branch on my nifi I get the weird
>>> > characters.
>>> >
>>> >
>>> > Some info on my setup :
>>> >
>>> > I'm sending commands to PutTCP using a GenerateFlowFile processor.
>>> > payload = ${literal('\r')}@COMMAND${literal('\r')}
>>> > PutTCP adds a delimier (\r\n)
>>> >
>>> > Regarding provenance. PutTCP adds the delimiter to the payload before
>>> > sending it to the TCP socket. That  delimiter is not in the flowfile
>>> > that is
>>> > put in provenance I think.
>>> >
>>> > Cannot reproduce the issue on my mac with my mock tcp server. But on the
>>> > actual environment (Linux CentOS) with the Moxa devices, there is a
>>> > clear
>>> > difference in behaviour. Will see if I can tunnel my through the devices
>>> >
>>> > All the info I have right now 
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Davy De Waele]

Correct about the delimiter.
Think character set is (was) automatically filled in with utf-8. (Never
noticed a default option)

On Wed, 20 Sep 2017 at 19:11, Bryan Bende  wrote:

> Thanks for the info and for digging into this..
>
> Just to confirm, in your PutTCP configuration, you are saying that you
> have "Outgoing Message Delimiter" set to \r\n  ?
>
> And do you have the "Character Set" property set to the default of "UTF-8"
> ?
>
> On Wed, Sep 20, 2017 at 12:59 PM, ddewaele  wrote:
> > I've put up a git repo that I use to test the PutTCP behavior.
> >
> > (
> https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1
> )
> >
> > What I've done.
> >
> > - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my base nifi
> > installation
> > - Noticed weird characters in the tcp responses
> > - Switched to 1.3.0. No more weird characters
> > - Isolated the PutTCP processor by creating a custom PutTCP (see github
> repo
> > below) to experiment (instead of the built-in PutTCP)
> > - Repo has 2 branches. master contains an "older" PutTCP processor from
> the
> > Nifi code base. With that one I don't have the weird characters (even on
> a
> > nifi 1.4.0-SNAPSHOT + your PR).
> > - If I use the dev_puttcp_1.4 branch on my nifi I get the weird
> characters.
> >
> >
> > Some info on my setup :
> >
> > I'm sending commands to PutTCP using a GenerateFlowFile processor.
> > payload = ${literal('\r')}@COMMAND${literal('\r')}
> > PutTCP adds a delimier (\r\n)
> >
> > Regarding provenance. PutTCP adds the delimiter to the payload before
> > sending it to the TCP socket. That  delimiter is not in the flowfile
> that is
> > put in provenance I think.
> >
> > Cannot reproduce the issue on my mac with my mock tcp server. But on the
> > actual environment (Linux CentOS) with the Moxa devices, there is a clear
> > difference in behaviour. Will see if I can tunnel my through the devices
> >
> > All the info I have right now 
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

Thanks for the info and for digging into this..

Just to confirm, in your PutTCP configuration, you are saying that you
have "Outgoing Message Delimiter" set to \r\n  ?

And do you have the "Character Set" property set to the default of "UTF-8" ?

On Wed, Sep 20, 2017 at 12:59 PM, ddewaele  wrote:
> I've put up a git repo that I use to test the PutTCP behavior.
>
> (https://github.com/ddewaele/com.ixortalk.tcp/compare/dev_puttcp_1.4?expand=1)
>
> What I've done.
>
> - Built Nifi 1.4.0-SNAPSHOT + your PR and used that as my base nifi
> installation
> - Noticed weird characters in the tcp responses
> - Switched to 1.3.0. No more weird characters
> - Isolated the PutTCP processor by creating a custom PutTCP (see github repo
> below) to experiment (instead of the built-in PutTCP)
> - Repo has 2 branches. master contains an "older" PutTCP processor from the
> Nifi code base. With that one I don't have the weird characters (even on a
> nifi 1.4.0-SNAPSHOT + your PR).
> - If I use the dev_puttcp_1.4 branch on my nifi I get the weird characters.
>
>
> Some info on my setup :
>
> I'm sending commands to PutTCP using a GenerateFlowFile processor.
> payload = ${literal('\r')}@COMMAND${literal('\r')}
> PutTCP adds a delimier (\r\n)
>
> Regarding provenance. PutTCP adds the delimiter to the payload before
> sending it to the TCP socket. That  delimiter is not in the flowfile that is
> put in provenance I think.
>
> Cannot reproduce the issue on my mac with my mock tcp server. But on the
> actual environment (Linux CentOS) with the Moxa devices, there is a clear
> difference in behaviour. Will see if I can tunnel my through the devices
>
> All the info I have right now 
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

Sounds good, interested to hear about it when you get a chance.

On Wed, Sep 20, 2017 at 12:32 PM, Davy De Waele  wrote:
> Having dinner now without laptop :) Have more updates. Isolated the problem
> in the puttcp class in the processor. Rest of the 1.4.0-snapshot seems fine.
>
> Will post more info after dinner.
>
> On Wed, 20 Sep 2017 at 18:27, Bryan Bende  wrote:
>>
>> Is the data good at other points in the flow before getting to PutTCP?
>>
>> If you are seeing those bytes in provenance, then they would have to
>> be in there before reaching PutTCP, because PutTCP doesn't modify the
>> content of the flow file.
>>
>> So I'm thinking something else in the flow before PutTCP is working
>> differently than before.
>>
>> On Wed, Sep 20, 2017 at 4:38 AM, ddewaele  wrote:
>> > Small update : No garbage / noise on 1.3.0 also so it must be in
>> > 1.4.0-SNAPSHOT.
>> >
>> > Noticed that the PutTCP processor has changed the way it processes
>> > incoming
>> > flowfiles. Might be related to that.
>> >
>> > Looked into the data provenance and noticed  3 bytes EF BF BD (in hex)
>> > when
>> > things go bad. (This seems to be the utf-8 encoding of the Unicode
>> > character
>> > U+FFFD REPLACEMENT CHARACTER.)
>> >
>> > So I'm guessing some kind of encoding / charset issue.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Davy De Waele]

Having dinner now without laptop :) Have more updates. Isolated the problem
in the puttcp class in the processor. Rest of the 1.4.0-snapshot seems fine.

Will post more info after dinner.

On Wed, 20 Sep 2017 at 18:27, Bryan Bende  wrote:

> Is the data good at other points in the flow before getting to PutTCP?
>
> If you are seeing those bytes in provenance, then they would have to
> be in there before reaching PutTCP, because PutTCP doesn't modify the
> content of the flow file.
>
> So I'm thinking something else in the flow before PutTCP is working
> differently than before.
>
> On Wed, Sep 20, 2017 at 4:38 AM, ddewaele  wrote:
> > Small update : No garbage / noise on 1.3.0 also so it must be in
> > 1.4.0-SNAPSHOT.
> >
> > Noticed that the PutTCP processor has changed the way it processes
> incoming
> > flowfiles. Might be related to that.
> >
> > Looked into the data provenance and noticed  3 bytes EF BF BD (in hex)
> when
> > things go bad. (This seems to be the utf-8 encoding of the Unicode
> character
> > U+FFFD REPLACEMENT CHARACTER.)
> >
> > So I'm guessing some kind of encoding / charset issue.
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>

Re: PutTCP connector not cleaning up dangling connections

[ddewaele]

Small update : No garbage / noise on 1.3.0 also so it must be in
1.4.0-SNAPSHOT.

Noticed that the PutTCP processor has changed the way it processes incoming
flowfiles. Might be related to that. 

Looked into the data provenance and noticed  3 bytes EF BF BD (in hex) when
things go bad. (This seems to be the utf-8 encoding of the Unicode character
U+FFFD REPLACEMENT CHARACTER.)

So I'm guessing some kind of encoding / charset issue.








--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[ddewaele]

I've let it run overnight on 1.4.0-SNAPSHOT. Didn't see any hanging
connections and after timeouts they were cleaned up.

However, I noticed something else (perhaps unrelated). About 40% of the
messages that we "get" from the tcp connection contained "noise / garbage"
and didn't pass their checksum. On 1.1.0 we never had that.

If I manually "put" data on the tcp connection (via a telnet session) to
trigger a response I don't see this "noise / garbage". So it seems to
originate  from PutTCP.

Any pointers ?  Going to investigate further today and check the detailed
release notes. (as I am coming from 1.1.0).





--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Joe Witt]

thanks davy - i'll get it reviewed and merged if you can give it a go
and verify it makes your case better.

Thanks

On Tue, Sep 19, 2017 at 1:57 PM, ddewaele  wrote:
> Hi,
>
> Trying it out now. forgot how long it takes to build :)
>
> Will give feedback here.
>
> Thx for the client port logging also ...  that is always useful for
> debugging perhaps we can check later in what way we can retrieve it in
> the timeout scenarios / standard close scenario
>
> Really hope this makes it into the 1.4.0 release.
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[ddewaele]

Hi,

Trying it out now. forgot how long it takes to build :)

Will give feedback here.

Thx for the client port logging also ...  that is always useful for
debugging perhaps we can check later in what way we can retrieve it in
the timeout scenarios / standard close scenario

Really hope this makes it into the 1.4.0 release.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Joe Witt]

Davy

If you could give the PR a try and see if it helps I'd be happy to
help get it reviewed and in for 1.4 if timing works out.

Thanks

On Mon, Sep 18, 2017 at 4:39 PM, Bryan Bende  wrote:
> Davy,
>
> I just pushed a second commit to the PR that will log the port from
> the local address of the socket being used by the sender, which I
> think is what you mean by the client port.
>
> If you turn on debug for PutTCP you will see something like...
>
> o.apache.nifi.processors.standard.PutTCP
> PutTCP[id=95463da0-015e-1000-8723-0a2baf6c832f] Connected to local
> port 57280
>
> I only performed the logging after successful connection because I'm
> not sure the ramifications of trying to obtain the local SocketAddress
> while finishConnect is still returning false, so I guess the logging
> wouldn't really help in the case where the exception was thrown.
>
> https://github.com/apache/nifi/pull/2159
>
> -Bryan
>
>
> On Mon, Sep 18, 2017 at 2:18 PM, ddewaele  wrote:
>> Thx a lot for the quick response. Looking forward to the PR and the
>> release :)
>>
>> Would this for example still make the 1.4.0 release ?
>>
>> It would also be very interesting to log client ports in debug mode 
>> don't know how easy that is with nio.
>>
>> There is Keep Alive Timeout = 2min specified on the Moxa, so it means that
>> the socket on the client (NiFi) is still responding to "keep alive" packets.
>> (makes sense I guess, as we would need to configure some kind of read
>> timeout on the moxa to kill off the client).
>>
>> I guess the fact that we don't see anything in the stack is because the
>> socket got established in non blocking mode so it is in ESTABLISHED mode but
>> nobody is around to do any processing on it.
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

Davy,

I just pushed a second commit to the PR that will log the port from
the local address of the socket being used by the sender, which I
think is what you mean by the client port.

If you turn on debug for PutTCP you will see something like...

o.apache.nifi.processors.standard.PutTCP
PutTCP[id=95463da0-015e-1000-8723-0a2baf6c832f] Connected to local
port 57280

I only performed the logging after successful connection because I'm
not sure the ramifications of trying to obtain the local SocketAddress
while finishConnect is still returning false, so I guess the logging
wouldn't really help in the case where the exception was thrown.

https://github.com/apache/nifi/pull/2159

-Bryan


On Mon, Sep 18, 2017 at 2:18 PM, ddewaele  wrote:
> Thx a lot for the quick response. Looking forward to the PR and the
> release :)
>
> Would this for example still make the 1.4.0 release ?
>
> It would also be very interesting to log client ports in debug mode 
> don't know how easy that is with nio.
>
> There is Keep Alive Timeout = 2min specified on the Moxa, so it means that
> the socket on the client (NiFi) is still responding to "keep alive" packets.
> (makes sense I guess, as we would need to configure some kind of read
> timeout on the moxa to kill off the client).
>
> I guess the fact that we don't see anything in the stack is because the
> socket got established in non blocking mode so it is in ESTABLISHED mode but
> nobody is around to do any processing on it.
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[ddewaele]

Thx a lot for the quick response. Looking forward to the PR and the
release :)

Would this for example still make the 1.4.0 release ?

It would also be very interesting to log client ports in debug mode 
don't know how easy that is with nio.

There is Keep Alive Timeout = 2min specified on the Moxa, so it means that
the socket on the client (NiFi) is still responding to "keep alive" packets.
(makes sense I guess, as we would need to configure some kind of read
timeout on the moxa to kill off the client).

I guess the fact that we don't see anything in the stack is because the
socket got established in non blocking mode so it is in ESTABLISHED mode but
nobody is around to do any processing on it.








--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[ddewaele]

Thx a lot for the quick response. Looking forward to the PR and the
release :)

Would this for example still make the 1.4.0 release ?

It would also be very interesting to log client ports in debug mode 
don't know how easy that is with nio.

There is Keep Alive Timeout = 2min specified on the Moxa, so it means that
the socket on the client (NiFi) is still responding to "keep alive" packets.
(makes sense I guess, as we would need to configure some kind of read
timeout on the moxa to kill off the client).

I guess the fact that we don't see anything in the stack is because the
socket got established in non blocking mode so it is in ESTABLISHED mode but
nobody is around to do any processing on it.








--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

I just created a JIRA and will put up a PR shortly:

https://issues.apache.org/jira/browse/NIFI-4391

The processor is catching the exception while attempting to obtain a
connection, and then logs an error and transfers to failure which is
where we see this message:

2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
o.apache.nifi.processors.standard.PutTCP
PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
and unable to create a new one, transferring

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/put/AbstractPutEventProcessor.java#L330-L347



On Mon, Sep 18, 2017 at 9:22 AM, Joe Witt  wrote:
> Good catch.  Can you please be sure to cover in a JIRA?
>
> That said, wouldn't we see that in the stack trace during the
> problematic condition?
>
> On Mon, Sep 18, 2017 at 9:16 AM, Bryan Bende  wrote:
>> The code in SocketChannelSender that Davy pointed out could definitely
>> be the problem...
>>
>> It makes a non-blocking channel and calls connect, then goes into a
>> loop waiting for finishConnect() to return true, but if that doesn't
>> happen before the configured timeout, then it throws an exception, but
>> it doesn't first close the channel, and the processor also doesn't
>> close the sender.
>>
>> As an example comparison, this code which is being used by PutTCP:
>>
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/put/sender/SocketChannelSender.java#L45-L78
>>
>> Should be doing something like this where it catches, closes, and rethrows:
>>
>> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/remote/io/socket/ssl/SSLSocketChannel.java#L140-L179
>>
>>
>> On Mon, Sep 18, 2017 at 9:09 AM, Joe Witt  wrote:
>>> Davy
>>>
>>> Interesting.  So in looking through the stack trace I don't see
>>> anything related to sockets nifi has initiated to another service and
>>> nothing for PutTCP.  I'm not saying that means there is nothing but
>>> the stack traces only show the custom GetTCP processors.
>>>
>>> You can use netstat to show open sockets from the nifi process.  Can
>>> you try that and share those?
>>>
>>> Does the NiFi UI show the processor as having stuck threads?  I'm
>>> guessing not since there is nothing in the stack traces.
>>>
>>> Thanks
>>>
>>> On Mon, Sep 18, 2017 at 1:54 AM, ddewaele  wrote:
 Stopping the processor doesn't cleanup the tcp connection. It remains
 ESTABLISHED.

 There are 2 ways of getting out of it (none of them are ideal).

 - Restarting Nifi
 - Restarting the Moxa serial ports

 I've dumped the output in the following gist :
 https://gist.github.com/ddewaele/83705003740674962c1e133fb617f68c

 The GetTCP processor you'll see in the thread dump also interacts with the
 moxa. It is a Netty based custom processor we created (because there was no
 GetTCP at the time). However, we log all interactions (including client
 ports) with this processor and all of them end up getting closed correctly.

 So the "hanging" connection originated from the built-in PutTCP processor.


 Joe Witt wrote
> If you stop the processor manually does it clean them up?
>
> When the connections appear stuck can you please get a thread dump?
>
> bin/nifi.sh dump
>
> The results end up in bootstrap.log.
>
> Thanks
> Joe
>
> On Sep 17, 2017 2:22 PM, "ddewaele" 

> ddewaele@

>  wrote:
>
>> We are using NiFi PutTCP processors to send messages to a number of Moxa
>> onCell ip gateway devices.
>>
>> These Moxa devices are running on a cellular network with not always the
>> most ideal connection. The Moxa only allows for a maximum of 2
>> simultaneous
>> client connections.
>>
>> What we notice is that although we specify connection / read timeouts on
>> both PutTCP and the Moxa, that sometimes a connection get "stuck". (In
>> the
>> moxa network monitoring we see 2 client sockets coming from PutTCP in the
>> ESTABLISHED state that never go away).
>>
>> This doesn't always happen, but often enough for it to be considered a
>> problem, as it requires a restart of the moxa ports to clear the
>> connections
>> (manual step). It typically happens when PutTCP experiences a Timeout.
>>
>> On the PutTCP processors we have the following settings :
>>
>> - Idle Connection Expiration : 30 seconds  (we've set this higher due to
>> bad
>> gprs connection)
>> - Timeout : 10 seconds (this is only used as a timeout for establishing
>> the
>> connection)
>>
>> On the Moxas we have
>>
>> - TCP alive 

Re: PutTCP connector not cleaning up dangling connections

[Joe Witt]

Good catch.  Can you please be sure to cover in a JIRA?

That said, wouldn't we see that in the stack trace during the
problematic condition?

On Mon, Sep 18, 2017 at 9:16 AM, Bryan Bende  wrote:
> The code in SocketChannelSender that Davy pointed out could definitely
> be the problem...
>
> It makes a non-blocking channel and calls connect, then goes into a
> loop waiting for finishConnect() to return true, but if that doesn't
> happen before the configured timeout, then it throws an exception, but
> it doesn't first close the channel, and the processor also doesn't
> close the sender.
>
> As an example comparison, this code which is being used by PutTCP:
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/put/sender/SocketChannelSender.java#L45-L78
>
> Should be doing something like this where it catches, closes, and rethrows:
>
> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/remote/io/socket/ssl/SSLSocketChannel.java#L140-L179
>
>
> On Mon, Sep 18, 2017 at 9:09 AM, Joe Witt  wrote:
>> Davy
>>
>> Interesting.  So in looking through the stack trace I don't see
>> anything related to sockets nifi has initiated to another service and
>> nothing for PutTCP.  I'm not saying that means there is nothing but
>> the stack traces only show the custom GetTCP processors.
>>
>> You can use netstat to show open sockets from the nifi process.  Can
>> you try that and share those?
>>
>> Does the NiFi UI show the processor as having stuck threads?  I'm
>> guessing not since there is nothing in the stack traces.
>>
>> Thanks
>>
>> On Mon, Sep 18, 2017 at 1:54 AM, ddewaele  wrote:
>>> Stopping the processor doesn't cleanup the tcp connection. It remains
>>> ESTABLISHED.
>>>
>>> There are 2 ways of getting out of it (none of them are ideal).
>>>
>>> - Restarting Nifi
>>> - Restarting the Moxa serial ports
>>>
>>> I've dumped the output in the following gist :
>>> https://gist.github.com/ddewaele/83705003740674962c1e133fb617f68c
>>>
>>> The GetTCP processor you'll see in the thread dump also interacts with the
>>> moxa. It is a Netty based custom processor we created (because there was no
>>> GetTCP at the time). However, we log all interactions (including client
>>> ports) with this processor and all of them end up getting closed correctly.
>>>
>>> So the "hanging" connection originated from the built-in PutTCP processor.
>>>
>>>
>>> Joe Witt wrote
 If you stop the processor manually does it clean them up?

 When the connections appear stuck can you please get a thread dump?

 bin/nifi.sh dump

 The results end up in bootstrap.log.

 Thanks
 Joe

 On Sep 17, 2017 2:22 PM, "ddewaele" 
>>>
 ddewaele@
>>>
  wrote:

> We are using NiFi PutTCP processors to send messages to a number of Moxa
> onCell ip gateway devices.
>
> These Moxa devices are running on a cellular network with not always the
> most ideal connection. The Moxa only allows for a maximum of 2
> simultaneous
> client connections.
>
> What we notice is that although we specify connection / read timeouts on
> both PutTCP and the Moxa, that sometimes a connection get "stuck". (In
> the
> moxa network monitoring we see 2 client sockets coming from PutTCP in the
> ESTABLISHED state that never go away).
>
> This doesn't always happen, but often enough for it to be considered a
> problem, as it requires a restart of the moxa ports to clear the
> connections
> (manual step). It typically happens when PutTCP experiences a Timeout.
>
> On the PutTCP processors we have the following settings :
>
> - Idle Connection Expiration : 30 seconds  (we've set this higher due to
> bad
> gprs connection)
> - Timeout : 10 seconds (this is only used as a timeout for establishing
> the
> connection)
>
> On the Moxas we have
>
> - TCP alive check time : 2min (this should force the Moxa to close the
> socket)
>
> Yet for some reason the connection remains established.
>
> Here's what I found out :
>
> On the moxa I noticed a connection (with client port 48440) that is in
> ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa
> I
> can see when the connection was established :
>
> 2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440
>
> I can track that down in Nifi via the logs (unfortunately PutTCP doesn't
> log
> client ports, but from the timestamp  I'm sure it's this connection :
>
> 2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.standard.PutTCP
> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
> creating a new 

Re: PutTCP connector not cleaning up dangling connections

[Bryan Bende]

The code in SocketChannelSender that Davy pointed out could definitely
be the problem...

It makes a non-blocking channel and calls connect, then goes into a
loop waiting for finishConnect() to return true, but if that doesn't
happen before the configured timeout, then it throws an exception, but
it doesn't first close the channel, and the processor also doesn't
close the sender.

As an example comparison, this code which is being used by PutTCP:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/put/sender/SocketChannelSender.java#L45-L78

Should be doing something like this where it catches, closes, and rethrows:

https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/remote/io/socket/ssl/SSLSocketChannel.java#L140-L179


On Mon, Sep 18, 2017 at 9:09 AM, Joe Witt  wrote:
> Davy
>
> Interesting.  So in looking through the stack trace I don't see
> anything related to sockets nifi has initiated to another service and
> nothing for PutTCP.  I'm not saying that means there is nothing but
> the stack traces only show the custom GetTCP processors.
>
> You can use netstat to show open sockets from the nifi process.  Can
> you try that and share those?
>
> Does the NiFi UI show the processor as having stuck threads?  I'm
> guessing not since there is nothing in the stack traces.
>
> Thanks
>
> On Mon, Sep 18, 2017 at 1:54 AM, ddewaele  wrote:
>> Stopping the processor doesn't cleanup the tcp connection. It remains
>> ESTABLISHED.
>>
>> There are 2 ways of getting out of it (none of them are ideal).
>>
>> - Restarting Nifi
>> - Restarting the Moxa serial ports
>>
>> I've dumped the output in the following gist :
>> https://gist.github.com/ddewaele/83705003740674962c1e133fb617f68c
>>
>> The GetTCP processor you'll see in the thread dump also interacts with the
>> moxa. It is a Netty based custom processor we created (because there was no
>> GetTCP at the time). However, we log all interactions (including client
>> ports) with this processor and all of them end up getting closed correctly.
>>
>> So the "hanging" connection originated from the built-in PutTCP processor.
>>
>>
>> Joe Witt wrote
>>> If you stop the processor manually does it clean them up?
>>>
>>> When the connections appear stuck can you please get a thread dump?
>>>
>>> bin/nifi.sh dump
>>>
>>> The results end up in bootstrap.log.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Sep 17, 2017 2:22 PM, "ddewaele" 
>>
>>> ddewaele@
>>
>>>  wrote:
>>>
 We are using NiFi PutTCP processors to send messages to a number of Moxa
 onCell ip gateway devices.

 These Moxa devices are running on a cellular network with not always the
 most ideal connection. The Moxa only allows for a maximum of 2
 simultaneous
 client connections.

 What we notice is that although we specify connection / read timeouts on
 both PutTCP and the Moxa, that sometimes a connection get "stuck". (In
 the
 moxa network monitoring we see 2 client sockets coming from PutTCP in the
 ESTABLISHED state that never go away).

 This doesn't always happen, but often enough for it to be considered a
 problem, as it requires a restart of the moxa ports to clear the
 connections
 (manual step). It typically happens when PutTCP experiences a Timeout.

 On the PutTCP processors we have the following settings :

 - Idle Connection Expiration : 30 seconds  (we've set this higher due to
 bad
 gprs connection)
 - Timeout : 10 seconds (this is only used as a timeout for establishing
 the
 connection)

 On the Moxas we have

 - TCP alive check time : 2min (this should force the Moxa to close the
 socket)

 Yet for some reason the connection remains established.

 Here's what I found out :

 On the moxa I noticed a connection (with client port 48440) that is in
 ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa
 I
 can see when the connection was established :

 2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440

 I can track that down in Nifi via the logs (unfortunately PutTCP doesn't
 log
 client ports, but from the timestamp  I'm sure it's this connection :

 2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
 o.apache.nifi.processors.standard.PutTCP
 PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
 creating a new one...
 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
 o.apache.nifi.processors.standard.PutTCP
 PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
 and unable to create a new one, transferring
 StandardFlowFileRecord[uuid=79f2a166-5211-4d2d-9275-03f0ce4d5b29,claim=
 StandardContentClaim

Re: PutTCP connector not cleaning up dangling connections

[Joe Witt]

Davy

Interesting.  So in looking through the stack trace I don't see
anything related to sockets nifi has initiated to another service and
nothing for PutTCP.  I'm not saying that means there is nothing but
the stack traces only show the custom GetTCP processors.

You can use netstat to show open sockets from the nifi process.  Can
you try that and share those?

Does the NiFi UI show the processor as having stuck threads?  I'm
guessing not since there is nothing in the stack traces.

Thanks

On Mon, Sep 18, 2017 at 1:54 AM, ddewaele  wrote:
> Stopping the processor doesn't cleanup the tcp connection. It remains
> ESTABLISHED.
>
> There are 2 ways of getting out of it (none of them are ideal).
>
> - Restarting Nifi
> - Restarting the Moxa serial ports
>
> I've dumped the output in the following gist :
> https://gist.github.com/ddewaele/83705003740674962c1e133fb617f68c
>
> The GetTCP processor you'll see in the thread dump also interacts with the
> moxa. It is a Netty based custom processor we created (because there was no
> GetTCP at the time). However, we log all interactions (including client
> ports) with this processor and all of them end up getting closed correctly.
>
> So the "hanging" connection originated from the built-in PutTCP processor.
>
>
> Joe Witt wrote
>> If you stop the processor manually does it clean them up?
>>
>> When the connections appear stuck can you please get a thread dump?
>>
>> bin/nifi.sh dump
>>
>> The results end up in bootstrap.log.
>>
>> Thanks
>> Joe
>>
>> On Sep 17, 2017 2:22 PM, "ddewaele" 
>
>> ddewaele@
>
>>  wrote:
>>
>>> We are using NiFi PutTCP processors to send messages to a number of Moxa
>>> onCell ip gateway devices.
>>>
>>> These Moxa devices are running on a cellular network with not always the
>>> most ideal connection. The Moxa only allows for a maximum of 2
>>> simultaneous
>>> client connections.
>>>
>>> What we notice is that although we specify connection / read timeouts on
>>> both PutTCP and the Moxa, that sometimes a connection get "stuck". (In
>>> the
>>> moxa network monitoring we see 2 client sockets coming from PutTCP in the
>>> ESTABLISHED state that never go away).
>>>
>>> This doesn't always happen, but often enough for it to be considered a
>>> problem, as it requires a restart of the moxa ports to clear the
>>> connections
>>> (manual step). It typically happens when PutTCP experiences a Timeout.
>>>
>>> On the PutTCP processors we have the following settings :
>>>
>>> - Idle Connection Expiration : 30 seconds  (we've set this higher due to
>>> bad
>>> gprs connection)
>>> - Timeout : 10 seconds (this is only used as a timeout for establishing
>>> the
>>> connection)
>>>
>>> On the Moxas we have
>>>
>>> - TCP alive check time : 2min (this should force the Moxa to close the
>>> socket)
>>>
>>> Yet for some reason the connection remains established.
>>>
>>> Here's what I found out :
>>>
>>> On the moxa I noticed a connection (with client port 48440) that is in
>>> ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa
>>> I
>>> can see when the connection was established :
>>>
>>> 2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440
>>>
>>> I can track that down in Nifi via the logs (unfortunately PutTCP doesn't
>>> log
>>> client ports, but from the timestamp  I'm sure it's this connection :
>>>
>>> 2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
>>> o.apache.nifi.processors.standard.PutTCP
>>> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
>>> creating a new one...
>>> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
>>> o.apache.nifi.processors.standard.PutTCP
>>> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
>>> and unable to create a new one, transferring
>>> StandardFlowFileRecord[uuid=79f2a166-5211-4d2d-9275-03f0ce4d5b29,claim=
>>> StandardContentClaim
>>> [resourceClaim=StandardResourceClaim[id=1505641210025-1,
>>> container=default,
>>> section=1], offset=84519, length=9],offset=0,name=
>>> 23934743676390659,size=9]
>>> to failure: java.net.SocketTimeoutException: Timed out connecting to
>>> 10.32.133.40:4001
>>> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
>>> o.apache.nifi.processors.standard.PutTCP
>>> java.net.SocketTimeoutException: Timed out connecting to
>>> 10.32.133.40:4001
>>> at
>>> org.apache.nifi.processor.util.put.sender.SocketChannelSender.open(
>>> SocketChannelSender.java:66)
>>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>>> at
>>> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.createSender(
>>> AbstractPutEventProcessor.java:312)
>>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>>> at
>>> org.apache.nifi.processors.standard.PutTCP.createSender(PutTCP.java:121)
>>> [nifi-standard-processors-1.1.0.jar:1.1.0]
>>> at
>>> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.
>>> 

Re: PutTCP connector not cleaning up dangling connections

[ddewaele]

Stopping the processor doesn't cleanup the tcp connection. It remains
ESTABLISHED.

There are 2 ways of getting out of it (none of them are ideal).

- Restarting Nifi
- Restarting the Moxa serial ports

I've dumped the output in the following gist :
https://gist.github.com/ddewaele/83705003740674962c1e133fb617f68c

The GetTCP processor you'll see in the thread dump also interacts with the
moxa. It is a Netty based custom processor we created (because there was no
GetTCP at the time). However, we log all interactions (including client
ports) with this processor and all of them end up getting closed correctly.

So the "hanging" connection originated from the built-in PutTCP processor.


Joe Witt wrote
> If you stop the processor manually does it clean them up?
> 
> When the connections appear stuck can you please get a thread dump?
> 
> bin/nifi.sh dump
> 
> The results end up in bootstrap.log.
> 
> Thanks
> Joe
> 
> On Sep 17, 2017 2:22 PM, "ddewaele" 

> ddewaele@

>  wrote:
> 
>> We are using NiFi PutTCP processors to send messages to a number of Moxa
>> onCell ip gateway devices.
>>
>> These Moxa devices are running on a cellular network with not always the
>> most ideal connection. The Moxa only allows for a maximum of 2
>> simultaneous
>> client connections.
>>
>> What we notice is that although we specify connection / read timeouts on
>> both PutTCP and the Moxa, that sometimes a connection get "stuck". (In
>> the
>> moxa network monitoring we see 2 client sockets coming from PutTCP in the
>> ESTABLISHED state that never go away).
>>
>> This doesn't always happen, but often enough for it to be considered a
>> problem, as it requires a restart of the moxa ports to clear the
>> connections
>> (manual step). It typically happens when PutTCP experiences a Timeout.
>>
>> On the PutTCP processors we have the following settings :
>>
>> - Idle Connection Expiration : 30 seconds  (we've set this higher due to
>> bad
>> gprs connection)
>> - Timeout : 10 seconds (this is only used as a timeout for establishing
>> the
>> connection)
>>
>> On the Moxas we have
>>
>> - TCP alive check time : 2min (this should force the Moxa to close the
>> socket)
>>
>> Yet for some reason the connection remains established.
>>
>> Here's what I found out :
>>
>> On the moxa I noticed a connection (with client port 48440) that is in
>> ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa
>> I
>> can see when the connection was established :
>>
>> 2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440
>>
>> I can track that down in Nifi via the logs (unfortunately PutTCP doesn't
>> log
>> client ports, but from the timestamp  I'm sure it's this connection :
>>
>> 2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
>> o.apache.nifi.processors.standard.PutTCP
>> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
>> creating a new one...
>> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
>> o.apache.nifi.processors.standard.PutTCP
>> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
>> and unable to create a new one, transferring
>> StandardFlowFileRecord[uuid=79f2a166-5211-4d2d-9275-03f0ce4d5b29,claim=
>> StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1505641210025-1,
>> container=default,
>> section=1], offset=84519, length=9],offset=0,name=
>> 23934743676390659,size=9]
>> to failure: java.net.SocketTimeoutException: Timed out connecting to
>> 10.32.133.40:4001
>> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
>> o.apache.nifi.processors.standard.PutTCP
>> java.net.SocketTimeoutException: Timed out connecting to
>> 10.32.133.40:4001
>> at
>> org.apache.nifi.processor.util.put.sender.SocketChannelSender.open(
>> SocketChannelSender.java:66)
>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.createSender(
>> AbstractPutEventProcessor.java:312)
>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processors.standard.PutTCP.createSender(PutTCP.java:121)
>> [nifi-standard-processors-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.
>> acquireSender(AbstractPutEventProcessor.java:334)
>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processors.standard.PutTCP.onTrigger(PutTCP.java:176)
>> [nifi-standard-processors-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(
>> StandardProcessorNode.java:1099)
>> [nifi-framework-core-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
>> ContinuallyRunProcessorTask.java:136)
>> [nifi-framework-core-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
>> ContinuallyRunProcessorTask.java:47)
>> [nifi-framework-core-1.1.0.jar:1.1.0]
>> at
>> 

Re: PutTCP connector not cleaning up dangling connections

[Joe Witt]

If you stop the processor manually does it clean them up?

When the connections appear stuck can you please get a thread dump?

bin/nifi.sh dump

The results end up in bootstrap.log.

Thanks
Joe

On Sep 17, 2017 2:22 PM, "ddewaele"  wrote:

> We are using NiFi PutTCP processors to send messages to a number of Moxa
> onCell ip gateway devices.
>
> These Moxa devices are running on a cellular network with not always the
> most ideal connection. The Moxa only allows for a maximum of 2 simultaneous
> client connections.
>
> What we notice is that although we specify connection / read timeouts on
> both PutTCP and the Moxa, that sometimes a connection get "stuck". (In the
> moxa network monitoring we see 2 client sockets coming from PutTCP in the
> ESTABLISHED state that never go away).
>
> This doesn't always happen, but often enough for it to be considered a
> problem, as it requires a restart of the moxa ports to clear the
> connections
> (manual step). It typically happens when PutTCP experiences a Timeout.
>
> On the PutTCP processors we have the following settings :
>
> - Idle Connection Expiration : 30 seconds  (we've set this higher due to
> bad
> gprs connection)
> - Timeout : 10 seconds (this is only used as a timeout for establishing the
> connection)
>
> On the Moxas we have
>
> - TCP alive check time : 2min (this should force the Moxa to close the
> socket)
>
> Yet for some reason the connection remains established.
>
> Here's what I found out :
>
> On the moxa I noticed a connection (with client port 48440) that is in
> ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa I
> can see when the connection was established :
>
> 2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440
>
> I can track that down in Nifi via the logs (unfortunately PutTCP doesn't
> log
> client ports, but from the timestamp  I'm sure it's this connection :
>
> 2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.standard.PutTCP
> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
> creating a new one...
> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.standard.PutTCP
> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
> and unable to create a new one, transferring
> StandardFlowFileRecord[uuid=79f2a166-5211-4d2d-9275-03f0ce4d5b29,claim=
> StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1505641210025-1,
> container=default,
> section=1], offset=84519, length=9],offset=0,name=
> 23934743676390659,size=9]
> to failure: java.net.SocketTimeoutException: Timed out connecting to
> 10.32.133.40:4001
> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.standard.PutTCP
> java.net.SocketTimeoutException: Timed out connecting to 10.32.133.40:4001
> at
> org.apache.nifi.processor.util.put.sender.SocketChannelSender.open(
> SocketChannelSender.java:66)
> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.createSender(
> AbstractPutEventProcessor.java:312)
> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processors.standard.PutTCP.createSender(PutTCP.java:121)
> [nifi-standard-processors-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.
> acquireSender(AbstractPutEventProcessor.java:334)
> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processors.standard.PutTCP.onTrigger(PutTCP.java:176)
> [nifi-standard-processors-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1099)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> [na:1.8.0_111]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_111]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>