[
https://issues.apache.org/jira/browse/KAFKA-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ekaterina Chernousova updated KAFKA-13427:
------------------------------------------
Summary: Max connections are not blocked (was: Max connections per
listener are not blocked)
> Max connections are not blocked
> -------------------------------
>
> Key: KAFKA-13427
> URL: https://issues.apache.org/jira/browse/KAFKA-13427
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 3.0.0
> Reporter: Ekaterina Chernousova
> Priority: Minor
>
> UPDATED 05/11/2021
> According to the documentation, new connections should be blocked when the
> max connections limit is reached except it's an inter-broker listener,
> however, it seems an external listener is treated the same way as an
> inter-broker listener.
> [http://kafka.apache.org/documentation/#brokerconfigs_max.connections]
> Shouldn't connections to external listeners be blocked/dropped when
> max.connections limit is reached instead of waiting for an available
> connection slot? Thanks!
>
> The following tests have been executed:
> *Test 1: Broker-wide limit*
> Comment: connections to external listeners are not being blocked/dropped the
> same way as it happens when per.ip broker-wide limit is reached, instead the
> system waits for an available connection slot.
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 10
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> {code}
> client logs when limit is reached (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 05:56:57,714] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 05:57:30,915] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 05:58:01,022] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 05:58:10,899] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 05:51:10,798 INFO [GroupCoordinator 0]: Assignment received from
> leader
> __strimzi-topic-operator-kstreams-45caeee4-1322-411d-a7f2-0bdcb9e38863-StreamThread-1-consumer-b6659f01-7981-4090-a9d3-89b9b80c340f
> for group __strimzi-topic-operator-kstreams for generation 1. The group has
> 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
> [data-plane-kafka-request-handler-7]
> 2021-11-05 05:56:09,169 INFO [Controller id=0] Processing automatic preferred
> replica leader election (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 05:56:09,169 TRACE [Controller id=0] Checking need to trigger auto
> leader balancing (kafka.controller.KafkaController) [controller-event-thread]
> 2021-11-05 05:56:09,173 DEBUG [Controller id=0] Topics not in preferred
> replica for broker 0 HashMap() (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 05:56:09,176 TRACE [Controller id=0] Leader imbalance ratio for
> broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
> {code}
> *Test 2: per.ip broker-wide limit*
> Comment: works as expected, however, it also blocks inter-broker connections
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 100
> max.connections.per.ip = 10
> max.connections.per.ip.overrides =
> {code}
> client logs when limit is reached (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 06:43:06,832] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:06,937] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:07,042] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:07,144] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:07,246] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 06:43:06,728 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:06,831 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:06,934 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,039 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,142 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,246 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,349 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> {code}
> *Test 3: listener-level limit*
> Comment: I assume connections should be blocked/dropped instead of waiting
> for an available connection slot
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 2147483647
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> listener.name.example-9095.max.connections=10
> {code}
> client logs when limit is reached (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 06:38:05,084] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:38:38,175] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:39:08,278] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:39:17,228] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 06:36:22,266 INFO [Controller id=0] Processing automatic preferred
> replica leader election (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:36:22,267 TRACE [Controller id=0] Checking need to trigger auto
> leader balancing (kafka.controller.KafkaController) [controller-event-thread]
> 2021-11-05 06:36:22,274 DEBUG [Controller id=0] Topics not in preferred
> replica for broker 0 HashMap() (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:36:22,274 TRACE [Controller id=0] Leader imbalance ratio for
> broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
> {code}
> *Test 4: Broker-wide limit with inter-broker connection*
> Comment: works as expected
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 10
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> {code}
> client that connects to an internal listener logs when limit is reached
> (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 06:56:51,602] WARN [Consumer
> clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
> Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
> disconnected (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:57:24,697] WARN [Consumer
> clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
> Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
> disconnected (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:57:57,797] WARN [Consumer
> clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
> Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
> disconnected (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 06:52:42,869 INFO [GroupCoordinator 0]: Assignment received from
> leader
> __strimzi-topic-operator-kstreams-ece4c6be-1043-467c-8785-872910af313f-StreamThread-1-consumer-1ed22ca1-b161-40ab-921f-994cdabe5f82
> for group __strimzi-topic-operator-kstreams for generation 1. The group has
> 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
> [data-plane-kafka-request-handler-4]
> 2021-11-05 06:57:40,509 INFO [Controller id=0] Processing automatic preferred
> replica leader election (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:57:40,509 TRACE [Controller id=0] Checking need to trigger auto
> leader balancing (kafka.controller.KafkaController) [controller-event-thread]
> 2021-11-05 06:57:40,518 DEBUG [Controller id=0] Topics not in preferred
> replica for broker 0 HashMap() (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:57:40,518 TRACE [Controller id=0] Leader imbalance ratio for
> broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
