Eric Chang created KAFKA-21587:
----------------------------------

             Summary: Change the default socket buffer configs on broker and 
clients to -1 (OS default)
                 Key: KAFKA-21587
                 URL: https://issues.apache.org/jira/browse/KAFKA-21587
             Project: Kafka
          Issue Type: Improvement
          Components: clients, core, network
            Reporter: Eric Chang


h3. Summary

Kafka sets fixed socket buffer sizes by default. On Linux, the kernel normally 
adjusts the buffer size of each TCP connection by itself (this is called 
auto-tuning). A fixed size turns this off for every Kafka connection. In my 
test, one consumer reads about 2 times faster when both the broker and the 
consumer use -1 (about 430 to 600 MB/s with the defaults, about 1000 to 1200 
MB/s with -1). With 1 ms of added delay it reads 14 times faster, and with 10 
ms 50 times faster. The cost is more kernel memory for connections where the 
consumer stops reading. I propose to make -1 the default.

h3. Proposal

# Change the defaults in the table in Background to -1.
# Remove {{socket.send.buffer.bytes=102400}} and 
{{socket.receive.buffer.bytes=102400}} from config/server.properties, 
config/broker.properties, config/controller.properties and 
docker/server.properties. These files set the values explicitly, so a change in 
the code alone does not reach users who start from them.
# Add a note to the upgrade docs.

h3. Background

||Config||Component||Default||
|socket.send.buffer.bytes|broker|102400|
|socket.receive.buffer.bytes|broker|102400|
|replica.socket.receive.buffer.bytes|broker|65536|
|send.buffer.bytes|clients|131072|
|receive.buffer.bytes|consumer, admin client|65536|
|receive.buffer.bytes|producer, Connect, Streams|32768|

Kafka passes these values to {{Socket.setSendBufferSize}} and 
{{Socket.setReceiveBufferSize}} (SO_SNDBUF and SO_RCVBUF). Without these 
options, Linux sizes the buffers of each connection by itself, between the 
limits in the {{tcp_rmem}} and {{tcp_wmem}} sysctls. It grows a buffer when the 
connection needs it. When an application sets these options, Linux keeps the 
buffer at that size and does not grow it. (Linux doubles the requested value, 
so 102400 becomes 204800.) The buffer size limits how much data can be in 
flight on the connection, so it limits throughput.

All these configs already accept -1 (KAFKA-724, KAFKA-3840 and follow-ups). 
With -1, Kafka does not call the setters and the kernel sizes the buffers. Some 
of Kafka's internal clients already use -1: the node-to-controller channel for 
both buffers, and the KRaft client and the replica fetcher for the send buffer.

The current values do not come from a measurement, as far as I can find. In the 
example config files, KAFKA-1531 (2014) changed both broker values from 1 MB to 
102400 (send) and 65536 (receive), and a later commit (49d7f8ee1cd, KAFKA-1723, 
2015) changed the receive value to 102400. I did not find the origin of the 100 
KB default in the code. The 64 KB consumer default comes from KAFKA-3463 
(2016), a mitigation for KAFKA-3135, which is still open.

h3. Measurement

Kafka trunk (commit e90d6f42c2), one broker (n2-standard-16) and one client 
(c4-standard-8) in one Google Cloud zone. Linux kernel 7.0.0-1017-gcp (Ubuntu 
24.04), with tcp_wmem max 4 MB and tcp_rmem max 32 MB (not changed). One 
consumer reads a topic with 12 partitions with {{kafka-consumer-perf-test}}. I 
added delay on the client with {{tc netem}}. Each case has 3 runs. The numbers 
are the median in MB/s. The first comment has the setup, the commands and more 
numbers.

||Added delay||Measured round-trip time||Both default||Only consumer -1||Only 
broker -1||Both -1||
|none|0.28 ms|427|731|436|1002|
|1 ms|1.3 to 1.5 ms|58.6|144|58.2|829|
|10 ms|10.3 ms|3.2|17.7|3.2|162|

* -1 on both sides is about 2.3 times faster with no added delay, 14 times with 
1 ms, and 50 times with 10 ms.
* Changing only the broker does nothing, because the consumer's receive buffer 
is the smaller one and stays the limit.
* Changing only the consumer gives about half of the gain with no added delay 
and about a tenth with 1 ms or 10 ms, because the limit moves to the broker's 
send buffer. Both must change.
* With the defaults, throughput falls faster than the round-trip time grows 
(about 7 and 133 times lower for a round-trip time about 5 and 37 times 
higher). I did not investigate why.
* The default case differs between days (427 to about 600 MB/s with no added 
delay) and I do not know why. All comparisons in this ticket are between runs 
on the same day.

The docs already say that high-latency connections may need larger socket 
buffers (docs/operations/datacenters.md).

h3. The kernel version matters with a fixed buffer

The Linux kernel commit 
[697a6c8cec03|https://github.com/torvalds/linux/commit/697a6c8cec03] ("tcp: 
increase the default TCP scaling ratio") names Kafka's default SO_RCVBUF=64k. 
It reports an application timeout after a kernel change made the window of a 
fixed buffer smaller. With a fixed buffer, Kafka depends on how each kernel 
version turns a buffer size into a window. With -1 the kernel can grow the 
buffer, so this matters much less.

h3. Cost

With -1 a connection can use more kernel memory, up to the tcp_wmem or tcp_rmem 
maximum. I measured the kernel TCP memory on the broker host ({{TCP: mem}} in 
/proc/net/sockstat) with many consumers in three states. "Stalled" means that 
each consumer read one fetch response, sent the next fetch, and then stopped 
reading. I used {{max.partition.fetch.bytes}} = 8 MB for this state, so that 
one response is larger than the largest send buffer. This is the worst case per 
connection.

||State||Consumers||Defaults on both sides||-1 on both sides||
|Idle (connected, no fetch)|2000|0 MB|0 MB|
|All reading|2000|307 MB|241 MB|
|Stalled|500|99 MB|1469 MB|

The idle row is the change from before the consumers connected (with -1 the 
counter was 3 MB before and after). The other rows are the reading itself.

* Idle connections cost the same. 2000 connections that all read also cost 
about the same, but each one was slow in this test (about 1.6 MB/s, because the 
client CPU was the limit). I did not measure many fast connections.
* A stalled fetch connection holds about 2.9 MB on the broker with -1 and about 
0.2 MB with the defaults. So a broker with many slow or stalled consumers uses 
much more kernel memory with -1. At this rate, about 1400 stalled fetch 
connections would reach the TCP memory pressure threshold of this broker host 
(the second value of the {{net.ipv4.tcp_mem}} sysctl, a threshold for all TCP 
sockets of the host together; about 4 GB on this host with 62 GB of memory). I 
did not measure that.
* The client side has a cost too, and its limit per connection is higher 
(tcp_rmem maximum 32 MB, against 4 MB for tcp_wmem). The client host with 500 
stalled consumers used 1962 MB with -1 (66 MB with the defaults). It reached 
the TCP memory pressure threshold of that host (about 1.9 GB; the host has 29 
GB of memory).
* I have no stalled number for 2000 consumers, because my test tool ran out of 
heap. See the first comment.

Users who want the old behaviour can set the old values explicitly.

h3. Not measured

* Produce direction (producer send buffer, broker receive buffer) and follower 
fetch. The same mechanism applies, but I have no numbers.
* Operating systems other than Linux.
* Only one broker, one consumer and one record size were tested.
* KAFKA-3135. I did not test if the pause described there can happen with -1. 
On Linux with -1, the receive buffer starts at the tcp_rmem default (131072 on 
the test VMs), which is the same size as today's 65536 after the kernel doubles 
it.
* Brokers in containers with a memory limit. Socket buffer memory counts toward 
the memory limit of the cgroup (cgroup v2).

h3. Questions

* Is there a reason to keep the fixed defaults that I have missed?
* Does this change need a KIP? I am happy to write one.

Related: KAFKA-16496 proposes -1 for the client receive.buffer.bytes only.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to