[
https://issues.apache.org/jira/browse/KAFKA-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18083114#comment-18083114
]
Yunseop Eom commented on KAFKA-19919:
-------------------------------------
I’d like to work on this if nobody is actively working on it.
I traced the blocking path to the broker-side non-GSSAPI branch in
`SaslServerAuthenticator#createSaslServer`. During SASL authentication from
`KafkaChannel.prepare()`, Kafka currently calls `Sasl.createSaslServer(...,
serverAddress().getHostName(), ...)`, and `InetAddress#getHostName()` may
trigger reverse DNS on the network thread.
This looks related to KAFKA-8562, but the path appears different: KAFKA-8562
addressed a client-side/SASL_SSL peer-host lookup, while this issue is the
broker-side `SaslServerAuthenticator` path. I would keep this scoped to the
broker/server path and leave client-side SASL and GSSAPI/Kerberos behavior
unchanged.
For Kafka’s built-in non-GSSAPI server mechanisms (`PLAIN`, `SCRAM-*`, and
`OAUTHBEARER`), the built-in server factories do not appear to use the
`serverName` argument. One possible fix is to avoid
`serverAddress().getHostName()` for those non-host-bound mechanisms, for
example by passing `null` as the Java SASL “unbound server” name.
The compatibility caveat is custom SASL providers, including providers
registered under built-in mechanism names, since mechanism name alone does not
prove provider identity. I’d add focused unit coverage for the broker-side
non-GSSAPI path and keep the GSSAPI/client paths unchanged.
Does this direction sound reasonable, or would maintainers prefer a different
replacement for `serverName`?
> Network Threads Blocked by Synchronous Reverse DNS Lookups During Connection
> Establishment
> ------------------------------------------------------------------------------------------
>
> Key: KAFKA-19919
> URL: https://issues.apache.org/jira/browse/KAFKA-19919
> Project: Kafka
> Issue Type: Bug
> Components: clients, core, network, security
> Affects Versions: 3.6.1
> Reporter: Nurzhan
> Priority: Major
>
> We had an issue with network threads described in
> [https://lists.apache.org/thread/hr26jkgsg243s8oyy3gq5y84vv9stodv.]
> In short, the problem was with intermittent low network thread idle percent
> due to high response send time, resulting in high response queue size.
> After some cycles of debugging and deploying the modified Kafka, we found out
> that the problem was in blocking DNS queries during the preparation of new
> connections.
> [https://github.com/apache/kafka/blob/be816b82d25370ceac697ccf7c88cea873e9b4e3/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L548]
> [https://github.com/apache/kafka/blob/be816b82d25370ceac697ccf7c88cea873e9b4e3/clients/src/main/java/org/apache/kafka/common/network/KafkaChannel.java#L174]
> [https://github.com/apache/kafka/blob/be816b82d25370ceac697ccf7c88cea873e9b4e3/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerAuthenticator.java#L209]
> Here, the KafkaChannel.prepare() call eventually results in calling
> serverAddress().getHostName(), which does the reverse DNS lookup on the
> network thread, blocking all the other connections assigned to the same
> network thread. The DNS query is pretty simple; it just resolves the broker
> hostname from IP, so we solved the problem of slow response send time by
> adding the DNS record in /etc/hosts. Though I'm not sure why there's such a
> query at all, adding the fixed record seems to fix the problem in our fairly
> static environment (I cannot guarantee that it will work in other
> environments). In our case, the network thread was blocked for 5 seconds due
> to the default DNS query timeout in Linux.
> We are writing this issue for those who come after because we couldn't find
> similar problems with Kafka on the web.
> In addition, maybe Kafka developers may consider some proposals:
> Adding networkThreadTimeNanos in
> [https://github.com/apache/kafka/blob/be816b82d25370ceac697ccf7c88cea873e9b4e3/core/src/main/scala/kafka/network/RequestChannel.scala#L201C30-L201C52]
> to the debug logs for each request, because it was hard to pinpoint the
> problem when the only supporting metric was response send time, which
> included the time to handle all connections during one selector poll
> Adding networkThreadTimeNanos to Selector.SelectorMetrics
> (https://github.com/apache/kafka/blob/be816b82d25370ceac697ccf7c88cea873e9b4e3/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L1121)
> Maybe cache the DNS queries
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
