[ 
https://issues.apache.org/jira/browse/NIFI-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147677#comment-18147677
 ] 

Jim Halfpenny commented on NIFI-16611:
--------------------------------------

I've built and tested this patch at it improves the behaviour and performance 
of load balancing in the event that a node becomes unreachable. Maximum 
flowfile queue size was reduced with the patch installed and flows between 
healthy nodes were not affected. The results of a comparative test are shown 
below.

 
||During the 5-minute blackhole of node 1's load balance port||Unpatched 
{{NiFi}}||Patched NiFi||
|Failed connect attempts per sending node|2, about 135 s apart|45, about 6 s 
apart|
|Error|{{ConnectException: Connection timed out}} (OS TCP 
timeout)|{{SocketTimeoutException: Connect timed out}} (the new 5 s timeout)|
|Max queue on the healthy nodes|about 450, growing until each OS timeout|52 at 
the start, then 25 or below|
|Transfers between the healthy nodes 0 and 2|Stalled for each 135 s connect, 
then a burst|Kept flowing|

 

> Add connect timeout to load balancing client sockets
> ----------------------------------------------------
>
>                 Key: NIFI-16611
>                 URL: https://issues.apache.org/jira/browse/NIFI-16611
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 2.12.0
>            Reporter: Jim Halfpenny
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{NioAsyncLoadBalanceClient.createChannel()}} connects in blocking mode via 
> {{socket.connect(address)}} without a timeout.  
> {{nifi.cluster.load.balance.comms.timeout}} is applied with 
> {{{}setSoTimeout{}}}, which does not cover connect. Against a peer that 
> silently drops packets, for example the previous address of a restarted node 
> in a dynamic environment such as Kubernetes, the call blocks for the OS TCP 
> connect timeout (about 127 seconds on Linux with default settings).
> {{NioAsyncLoadBalanceClientTask}} iterates serially over all registered 
> clients. While one client is blocked in connect, load-balanced transfers to 
> every other node served by that task thread are stalled. A failed connect 
> only penalizes the client for 1 second, so the stall repeats for as long as 
> the peer address is unreachable and the node is still considered connected.
> This is the same failure mode as NIFI-16547 in the cluster protocol sockets.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
  • [jira] [Commented] (NIFI... Jim Halfpenny (Jira) via issues

Reply via email to