[
https://issues.apache.org/jira/browse/NIFI-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148174#comment-18148174
]
Jim Halfpenny commented on NIFI-16618:
--------------------------------------
*Cluster test (3 nodes, SDP 26.11.0 operators, NiFI 2.12.0)*
To make reverse lookups slow I set CoreDNS to drop every PTR query for the pod
network. A reverse lookup of a pod IP then took 10s in both clusters (5s
resolver timeout × 2 attempts). With the patch clustering worked even with the
dropped DNS PTR queries, but without the patch clustering was broken for the
whole duration of the test.
|| ||Stock {{nifi:2.12.0-stackable26.11.0}}||Patched||
|Cluster formed|*No.* Nodes 0 and 1 never joined (30 failed attempts each in 5
min)|Yes, all 3 nodes connected within seconds|
|Joining node error|{{SocketTimeoutException: Read timed out}} on
{{CONNECTION_REQUEST}}|None|
|Coordinator error|59 × {{Failed processing protocol message … Broken
pipe}}|None|
|Coordinator request handling|Spent 10 s on the reverse lookup before reading
each remote request|239 requests in 5 min, avg 44 ms, max 85 ms|
|Node status (polled every 30 s for 5 min)|Never formed|All 3 CONNECTED
throughout|
> Avoid reverse DNS lookups in cluster protocol listener
> ------------------------------------------------------
>
> Key: NIFI-16618
> URL: https://issues.apache.org/jira/browse/NIFI-16618
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.12.0
> Reporter: Jim Halfpenny
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{SocketProtocolListener.dispatchRequest()}} calls
> {{socket.getInetAddress().getHostName()}} for every inbound cluster protocol
> connection before reading the request. For an accepted socket this performs a
> reverse DNS (PTR) lookup. The result is only used in log messages.
> When PTR lookups are slow, every protocol message waits for the resolver
> timeout before processing starts. This includes heartbeats to the Cluster
> Coordinator. The JVM caches negative lookups for 10 seconds by default, so
> the delay recurs after this period. If heartbeat processing falls behind for
> long enough, nodes are disconnected for missed heartbeats.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)