Re: PutTCP connector not cleaning up dangling connections

[ddewaele]

Stopping the processor doesn't cleanup the tcp connection. It remains
ESTABLISHED.

There are 2 ways of getting out of it (none of them are ideal).

- Restarting Nifi
- Restarting the Moxa serial ports

I've dumped the output in the following gist :
https://gist.github.com/ddewaele/83705003740674962c1e133fb617f68c

The GetTCP processor you'll see in the thread dump also interacts with the
moxa. It is a Netty based custom processor we created (because there was no
GetTCP at the time). However, we log all interactions (including client
ports) with this processor and all of them end up getting closed correctly.

So the "hanging" connection originated from the built-in PutTCP processor.


Joe Witt wrote
> If you stop the processor manually does it clean them up?
> 
> When the connections appear stuck can you please get a thread dump?
> 
> bin/nifi.sh dump
> 
> The results end up in bootstrap.log.
> 
> Thanks
> Joe
> 
> On Sep 17, 2017 2:22 PM, "ddewaele" 

> ddewaele@

>  wrote:
> 
>> We are using NiFi PutTCP processors to send messages to a number of Moxa
>> onCell ip gateway devices.
>>
>> These Moxa devices are running on a cellular network with not always the
>> most ideal connection. The Moxa only allows for a maximum of 2
>> simultaneous
>> client connections.
>>
>> What we notice is that although we specify connection / read timeouts on
>> both PutTCP and the Moxa, that sometimes a connection get "stuck". (In
>> the
>> moxa network monitoring we see 2 client sockets coming from PutTCP in the
>> ESTABLISHED state that never go away).
>>
>> This doesn't always happen, but often enough for it to be considered a
>> problem, as it requires a restart of the moxa ports to clear the
>> connections
>> (manual step). It typically happens when PutTCP experiences a Timeout.
>>
>> On the PutTCP processors we have the following settings :
>>
>> - Idle Connection Expiration : 30 seconds  (we've set this higher due to
>> bad
>> gprs connection)
>> - Timeout : 10 seconds (this is only used as a timeout for establishing
>> the
>> connection)
>>
>> On the Moxas we have
>>
>> - TCP alive check time : 2min (this should force the Moxa to close the
>> socket)
>>
>> Yet for some reason the connection remains established.
>>
>> Here's what I found out :
>>
>> On the moxa I noticed a connection (with client port 48440) that is in
>> ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa
>> I
>> can see when the connection was established :
>>
>> 2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440
>>
>> I can track that down in Nifi via the logs (unfortunately PutTCP doesn't
>> log
>> client ports, but from the timestamp  I'm sure it's this connection :
>>
>> 2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
>> o.apache.nifi.processors.standard.PutTCP
>> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
>> creating a new one...
>> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
>> o.apache.nifi.processors.standard.PutTCP
>> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
>> and unable to create a new one, transferring
>> StandardFlowFileRecord[uuid=79f2a166-5211-4d2d-9275-03f0ce4d5b29,claim=
>> StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1505641210025-1,
>> container=default,
>> section=1], offset=84519, length=9],offset=0,name=
>> 23934743676390659,size=9]
>> to failure: java.net.SocketTimeoutException: Timed out connecting to
>> 10.32.133.40:4001
>> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
>> o.apache.nifi.processors.standard.PutTCP
>> java.net.SocketTimeoutException: Timed out connecting to
>> 10.32.133.40:4001
>> at
>> org.apache.nifi.processor.util.put.sender.SocketChannelSender.open(
>> SocketChannelSender.java:66)
>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.createSender(
>> AbstractPutEventProcessor.java:312)
>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processors.standard.PutTCP.createSender(PutTCP.java:121)
>> [nifi-standard-processors-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.
>> acquireSender(AbstractPutEventProcessor.java:334)
>> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.processors.standard.PutTCP.onTrigger(PutTCP.java:176)
>> [nifi-standard-processors-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(
>> StandardProcessorNode.java:1099)
>> [nifi-framework-core-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
>> ContinuallyRunProcessorTask.java:136)
>> [nifi-framework-core-1.1.0.jar:1.1.0]
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
>> ContinuallyRunProcessorTask.java:47)
>> [nifi-framework-core-1.1.0.jar:1.1.0]
>> at
>> 

Re: PutTCP connector not cleaning up dangling connections

[Joe Witt]

If you stop the processor manually does it clean them up?

When the connections appear stuck can you please get a thread dump?

bin/nifi.sh dump

The results end up in bootstrap.log.

Thanks
Joe

On Sep 17, 2017 2:22 PM, "ddewaele"  wrote:

> We are using NiFi PutTCP processors to send messages to a number of Moxa
> onCell ip gateway devices.
>
> These Moxa devices are running on a cellular network with not always the
> most ideal connection. The Moxa only allows for a maximum of 2 simultaneous
> client connections.
>
> What we notice is that although we specify connection / read timeouts on
> both PutTCP and the Moxa, that sometimes a connection get "stuck". (In the
> moxa network monitoring we see 2 client sockets coming from PutTCP in the
> ESTABLISHED state that never go away).
>
> This doesn't always happen, but often enough for it to be considered a
> problem, as it requires a restart of the moxa ports to clear the
> connections
> (manual step). It typically happens when PutTCP experiences a Timeout.
>
> On the PutTCP processors we have the following settings :
>
> - Idle Connection Expiration : 30 seconds  (we've set this higher due to
> bad
> gprs connection)
> - Timeout : 10 seconds (this is only used as a timeout for establishing the
> connection)
>
> On the Moxas we have
>
> - TCP alive check time : 2min (this should force the Moxa to close the
> socket)
>
> Yet for some reason the connection remains established.
>
> Here's what I found out :
>
> On the moxa I noticed a connection (with client port 48440) that is in
> ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa I
> can see when the connection was established :
>
> 2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440
>
> I can track that down in Nifi via the logs (unfortunately PutTCP doesn't
> log
> client ports, but from the timestamp  I'm sure it's this connection :
>
> 2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.standard.PutTCP
> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
> creating a new one...
> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.standard.PutTCP
> PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
> and unable to create a new one, transferring
> StandardFlowFileRecord[uuid=79f2a166-5211-4d2d-9275-03f0ce4d5b29,claim=
> StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1505641210025-1,
> container=default,
> section=1], offset=84519, length=9],offset=0,name=
> 23934743676390659,size=9]
> to failure: java.net.SocketTimeoutException: Timed out connecting to
> 10.32.133.40:4001
> 2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
> o.apache.nifi.processors.standard.PutTCP
> java.net.SocketTimeoutException: Timed out connecting to 10.32.133.40:4001
> at
> org.apache.nifi.processor.util.put.sender.SocketChannelSender.open(
> SocketChannelSender.java:66)
> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.createSender(
> AbstractPutEventProcessor.java:312)
> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processors.standard.PutTCP.createSender(PutTCP.java:121)
> [nifi-standard-processors-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processor.util.put.AbstractPutEventProcessor.
> acquireSender(AbstractPutEventProcessor.java:334)
> ~[nifi-processor-utils-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.processors.standard.PutTCP.onTrigger(PutTCP.java:176)
> [nifi-standard-processors-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1099)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.1.0.jar:1.1.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> [na:1.8.0_111]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_111]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> 

Re: PutEmail with Multiple Attachements

[Mika Borner]

Thanks for your reply

Customer was ok with the Zip approach, otherwise I would have used the 
ExecuteProcess processor to run a custom script.


Mika>


On 09/15/2017 07:39 PM, Joe Witt wrote:

Mika,

Could it be ok for your case to use MergeContent to package data
together in some archive format like Zip/Tar?  If yes then you can
just add a tar/zip to your email.

Otherwise, the changes necessary to PutEmail for supporting multiple
attachments would be quite involved.

Thanks

On Wed, Sep 13, 2017 at 12:26 PM, Mika Borner  wrote:

Hi

I have a use case where I need to attach two files to a mail message.

As the PutEmail processor only handles one flowfile as an attachment at a
time, I'm trying to compose the multipart message by myself.

I have Base64 encoded the attachment and added boundaries, and then added
the data into the message attribute, but now I'm unable to set the Content
Type to "multipart/mixed; boundary=", which would be needed.

I'm getting an exception:

2017-09-13 18:12:08,850 ERROR [Timer-Driven Process Thread-10]
o.a.nifi.processors.standard.PutEmail
PutEmail[id=015e1000-5cef-1d09-a660-e1ef3a8825ab] Failed to send email for
StandardFlowFileRecord[uuid=2beb9198-06d6-4a68-8456-aaed0c6fc89a,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1505288584837-1, container=default,
section=1], offset=813176,
length=197],offset=0,name=33321931515708,size=197]: MIME part of type
"multipart/mixed; boundary=attachement" contains object of type
java.lang.String instead of MimeMultipart; routing to failure:
javax.mail.MessagingException: MIME part of type "multipart/mixed;
boundary=attachement" contains object of type java.lang.String instead of
MimeMultipart
javax.mail.MessagingException: MIME part of type "multipart/mixed;
boundary=attachement" contains object of type java.lang.String instead of
MimeMultipart.

Any ideas? Or any other suggestions how I can send multiple mail
attachements with NiFi?

Thanks!

Mika>

PutTCP connector not cleaning up dangling connections

[ddewaele]

We are using NiFi PutTCP processors to send messages to a number of Moxa
onCell ip gateway devices.

These Moxa devices are running on a cellular network with not always the
most ideal connection. The Moxa only allows for a maximum of 2 simultaneous
client connections.

What we notice is that although we specify connection / read timeouts on
both PutTCP and the Moxa, that sometimes a connection get "stuck". (In the
moxa network monitoring we see 2 client sockets coming from PutTCP in the
ESTABLISHED state that never go away).

This doesn't always happen, but often enough for it to be considered a
problem, as it requires a restart of the moxa ports to clear the connections
(manual step). It typically happens when PutTCP experiences a Timeout.

On the PutTCP processors we have the following settings :

- Idle Connection Expiration : 30 seconds  (we've set this higher due to bad
gprs connection)
- Timeout : 10 seconds (this is only used as a timeout for establishing the
connection)

On the Moxas we have

- TCP alive check time : 2min (this should force the Moxa to close the
socket)

Yet for some reason the connection remains established.

Here's what I found out :

On the moxa I noticed a connection (with client port 48440) that is in
ESTABLISHED mode for 4+ hours. (blocking other connections). On the Moxa I
can see when the connection was established :

2017/09/17 14:20:29 [OpMode] Port01 Connect 10.192.2.90:48440

I can track that down in Nifi via the logs (unfortunately PutTCP doesn't log
client ports, but from the timestamp  I'm sure it's this connection :

2017-09-17 14:20:10,837 DEBUG [Timer-Driven Process Thread-10]
o.apache.nifi.processors.standard.PutTCP
PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
creating a new one...
2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
o.apache.nifi.processors.standard.PutTCP
PutTCP[id=80231a39-1008-1159-a6fa-1f9e3751d608] No available connections,
and unable to create a new one, transferring
StandardFlowFileRecord[uuid=79f2a166-5211-4d2d-9275-03f0ce4d5b29,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1505641210025-1, container=default,
section=1], offset=84519, length=9],offset=0,name=23934743676390659,size=9]
to failure: java.net.SocketTimeoutException: Timed out connecting to
10.32.133.40:4001
2017-09-17 14:20:20,860 ERROR [Timer-Driven Process Thread-10]
o.apache.nifi.processors.standard.PutTCP 
java.net.SocketTimeoutException: Timed out connecting to 10.32.133.40:4001
at
org.apache.nifi.processor.util.put.sender.SocketChannelSender.open(SocketChannelSender.java:66)
~[nifi-processor-utils-1.1.0.jar:1.1.0]
at
org.apache.nifi.processor.util.put.AbstractPutEventProcessor.createSender(AbstractPutEventProcessor.java:312)
~[nifi-processor-utils-1.1.0.jar:1.1.0]
at
org.apache.nifi.processors.standard.PutTCP.createSender(PutTCP.java:121)
[nifi-standard-processors-1.1.0.jar:1.1.0]
at
org.apache.nifi.processor.util.put.AbstractPutEventProcessor.acquireSender(AbstractPutEventProcessor.java:334)
~[nifi-processor-utils-1.1.0.jar:1.1.0]
at
org.apache.nifi.processors.standard.PutTCP.onTrigger(PutTCP.java:176)
[nifi-standard-processors-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_111]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_111]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]


On an OS level I indeed see the TCP connection originated from Nifi :

netstat -tn | grep 48440

tcp  711  0 10.192.2.90:48440   10.32.133.40:4001  
ESTABLISHED


lsof -i TCP:48440

COMMAND  PID USER   FD   TYPEDEVICE SIZE/OFF NODE NAME
java3424 root 1864u  IPv4 404675057  0t0  TCP
NifiServer:48440->10.32.133.40:newoak (ESTABLISHED)

ps -ef | grep 3424

root  3424  3390  8 

Manual release of flows

[joe harvyy]

Hi,

I have a use case where flow files should be blocked before moving to the
next step. A manual action is needed to release these flow files. This
manual action will be an HTTP post that an admin will be sending to NiFi
through a simple web UI.

How can I implement this behavior? Can I use Wait/Notify in this case or is
there a better approach?

Thanks
J.