[
https://issues.apache.org/jira/browse/NIFI-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Halfpenny updated NIFI-16611:
---------------------------------
Description:
{{NioAsyncLoadBalanceClient.createChannel()}} connects in blocking mode via
{{socket.connect(address)}} without a timeout.
{{nifi.cluster.load.balance.comms.timeout}} is applied with
{{{}setSoTimeout{}}}, which does not cover connect. Against a peer that
silently drops packets, for example the previous address of a restarted node in
a dynamic environment such as Kubernetes, the call blocks for the OS TCP
connect timeout (about 127 seconds on Linux with default settings).
{{NioAsyncLoadBalanceClientTask}} iterates serially over all registered
clients. While one client is blocked in connect, load-balanced transfers to
every other node served by that task thread are stalled. A failed connect only
penalizes the client for 1 second, so the stall repeats for as long as the peer
address is unreachable and the node is still considered connected.
This is the same failure mode as NIFI-16547 in the cluster protocol sockets.
was:
{{{{{{NioAsyncLoadBalanceClient.createChannel()}}}}}} connects in blocking mode
via {{socket.connect(address)}} without a timeout.
{{nifi.cluster.load.balance.comms.timeout}} is applied with
{{{}setSoTimeout{}}}, which does not cover connect. Against a peer that
silently drops packets, for example the previous address of a restarted node in
a dynamic environment such as Kubernetes, the call blocks for the OS TCP
connect timeout (about 127 seconds on Linux with default settings).
{{NioAsyncLoadBalanceClientTask}} iterates serially over all registered
clients. While one client is blocked in connect, load-balanced transfers to
every other node served by that task thread are stalled. A failed connect only
penalizes the client for 1 second, so the stall repeats for as long as the peer
address is unreachable and the node is still considered connected.
This is the same failure mode as NIFI-16547 in the cluster protocol sockets.
> Add connect timeout to load balancing client sockets
> ----------------------------------------------------
>
> Key: NIFI-16611
> URL: https://issues.apache.org/jira/browse/NIFI-16611
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.12.0
> Reporter: Jim Halfpenny
> Priority: Major
>
> {{NioAsyncLoadBalanceClient.createChannel()}} connects in blocking mode via
> {{socket.connect(address)}} without a timeout.
> {{nifi.cluster.load.balance.comms.timeout}} is applied with
> {{{}setSoTimeout{}}}, which does not cover connect. Against a peer that
> silently drops packets, for example the previous address of a restarted node
> in a dynamic environment such as Kubernetes, the call blocks for the OS TCP
> connect timeout (about 127 seconds on Linux with default settings).
> {{NioAsyncLoadBalanceClientTask}} iterates serially over all registered
> clients. While one client is blocked in connect, load-balanced transfers to
> every other node served by that task thread are stalled. A failed connect
> only penalizes the client for 1 second, so the stall repeats for as long as
> the peer address is unreachable and the node is still considered connected.
> This is the same failure mode as NIFI-16547 in the cluster protocol sockets.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)