[ 
https://issues.apache.org/jira/browse/NIFI-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148174#comment-18148174
 ] 

Jim Halfpenny commented on NIFI-16618:
--------------------------------------

*Cluster test (3 nodes, SDP 26.11.0 operators, NiFI 2.12.0)*
To make reverse lookups slow I set CoreDNS to drop every PTR query for the pod 
network. A reverse lookup of a pod IP then took 10s in both clusters (5s 
resolver timeout × 2 attempts). With the patch clustering worked even with the 
dropped DNS PTR queries, but without the patch clustering was broken for the 
whole duration of the test.


|| ||Stock {{nifi:2.12.0-stackable26.11.0}}||Patched||
|Cluster formed|*No.* Nodes 0 and 1 never joined (30 failed attempts each in 5 
min)|Yes, all 3 nodes connected within seconds|
|Joining node error|{{SocketTimeoutException: Read timed out}} on 
{{CONNECTION_REQUEST}}|None|
|Coordinator error|59 × {{Failed processing protocol message … Broken 
pipe}}|None|
|Coordinator request handling|Spent 10 s on the reverse lookup before reading 
each remote request|239 requests in 5 min, avg 44 ms, max 85 ms|
|Node status (polled every 30 s for 5 min)|Never formed|All 3 CONNECTED 
throughout|

 

> Avoid reverse DNS lookups in cluster protocol listener
> ------------------------------------------------------
>
>                 Key: NIFI-16618
>                 URL: https://issues.apache.org/jira/browse/NIFI-16618
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 2.12.0
>            Reporter: Jim Halfpenny
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{SocketProtocolListener.dispatchRequest()}} calls 
> {{socket.getInetAddress().getHostName()}} for every inbound cluster protocol 
> connection before reading the request. For an accepted socket this performs a 
> reverse DNS (PTR) lookup. The result is only used in log messages.
> When PTR lookups are slow, every protocol message waits for the resolver 
> timeout before processing starts. This includes heartbeats to the Cluster 
> Coordinator. The JVM caches negative lookups for 10 seconds by default, so 
> the delay recurs after this period. If heartbeat processing falls behind for 
> long enough, nodes are disconnected for missed heartbeats.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to