[
https://issues.apache.org/jira/browse/KAFKA-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Iatsuk updated KAFKA-12481:
----------------------------------
Description:
h3. Problem
A large number of topic-partitions on one broker causes a burst of the host's
packets/sec metric. Most packets are small inter-broker (replication fetch)
responses of 40-160 bytes. A traffic shaper in the cloud stops coping with such
a load and causes service degradation. The original measurements from 2021 are
below.
h3. Root cause
A FetchResponse is built by {{SendBuilder}} as a {{MultiRecordsSend}}: for
every partition there is a small header buffer and the records.
{{MultiRecordsSend.writeTo()}} writes every part with its own system call, and
because {{TCP_NODELAY}} is always set, every call leaves as its own TCP
segment. With SSL it was even more, because {{SslTransportLayer}} wrapped every
buffer into its own TLS record and wrote it alone, also for the requests of the
clients.
h3. Fix
With SSL the fix is inside {{SslTransportLayer}}, without a new config. PR:
https://github.com/apache/kafka/pull/24170
* {{SslTransportLayer}} gives all buffers of a write to {{SSLEngine.wrap()}},
so they go into one TLS record.
* {{KafkaChannel.write()}} opens a write batch on the transport layer around
the write of a send. In a batch {{SslTransportLayer}} keeps the TLS records in
its buffer and writes them to the socket with one call when the batch ends.
Jun Rao proposed both in the review. PLAINTEXT is not changed by this PR, see
"Open" below.
h3. Results on trunk (cfcdfb7654)
3 brokers on 3 VMs of a Kubernetes cluster, SSL. The table shows trunk and the
difference to trunk with a 95% interval:
||Load||Packets/s on trunk||Packets/s||Bytes on the wire||Broker CPU||
|5 topics x 600 partitions, RF=3, 3,000 records/s of 100 bytes,
acks=all|37,938|−79.3% ± 0.2%|−34.4% ± 0.2%|−5.2% ± 1.5%|
|20 MB/s into 30 partitions, RF=3, 2 consumers|88,485|−12.3% ± 0.3%|−0.6% ±
0.2%|−1.5% ± 1.7%|
With one record per request (one broker, loopback) the maximum throughput with
SSL is 10% higher (35,511 -> 39,277 records/s). With PLAINTEXT nothing changes.
The details are in the notes of the PR.
h3. History of the approaches
# 2021: a config to turn {{TCP_NODELAY}} off (KIP-723, PR #10333). It was
dropped. With Nagle's algorithm the end-to-end latency with acks=all is 3 times
higher (p50 18 -> 57 ms on current trunk), as Colin McCabe wrote in the KIP
discussion. KIP-723 is marked as discarded.
# 2026-11, first version of PR #24170: {{KafkaChannel.write()}} turned
{{TCP_NODELAY}} off while it wrote a send, and on again after the write. Linux
sends the queued data at once when {{TCP_NODELAY}} is turned on, so the parts
left as full segments and nothing waited for an ACK. It removed 56% of the
packets with PLAINTEXT and 73% with SSL, with the same CPU and latency. It was
removed after the review for two reasons. It depends on the Linux TCP stack,
and other stacks that report Linux (gVisor, kernel-bypass libraries) may behave
differently. And it cannot be turned off without a new config, which needs a
KIP.
# 2026-11, current version: the fix inside {{SslTransportLayer}} described
above. It does not depend on the kernel.
h3. Open
With PLAINTEXT {{MultiRecordsSend}} still writes every part with its own system
call. An idea for a follow-up is the same write batch in user space: small
parts are collected and written with one call, large parts still go with
zero-copy.
h3. Links
* KIP-723:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-723%3A+Add+socket.tcp.no.delay+property+to+Kafka+Config
* dev@ discussion 2021:
https://lists.apache.org/thread/d5vwzvhyy1f7wwrxpcy173trlyx0c6vk
* dev@ discussion 2025:
https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3
* dev@ discussion 2026:
https://lists.apache.org/thread/ymvfjg6fp7k2y1pwds268vv39rq5ohyx
----
h3. Original report (2021)
*What to do?*
Add _socket.nagle.disable_ parameter to Apache Kafka config like in
[librdkafka|https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md].
*What reason of this improvement?*
A large number of topic-partitions on one broker causes burst of host's
packets/sec metric. The traffic shaper in the cloud ceases to cope with such a
load and causes service degradation.
*How to reproduce?*
# Create Kafka Cluster with 4 brokers. Amount of packet/sec is ~120.
# Add 100 topics with 100 partitions each and replication factor = 3. It is 30k
topic-partitions in total. Amount of packet/sec is ~15k.
{code:java}
import os
for i in range(100):
print(f"create topic 'flower{i}'... ", end="")
cmd = "kafka-topics.sh --create --bootstrap-server {} --topic {} --partitions
{} --replication-factor {}".format("databus.andrei-iatsuk.ec.odkl.ru:9092",
f"flower{i}", 100, 3)
code = os.system(cmd)
print("ok" if code == 0 else "error")
{code}
!Screenshot 2021-03-16 at 21.05.03.png!
# Generate server load by launching next script in 4 terminals. Amount of
packet/sec is ~130k.
{code:java}
import time
from pykafka import KafkaClient
client = KafkaClient(hosts="databus.andrei-iatsuk.ec.odkl.ru:9092")
while True:
for i in range(100):
print(f"sent message to 'flower{i}'")
with client.topics[f"flower{i}"].get_sync_producer() as producer:
for j in range(1000):
producer.produce(str.encode(f'test message {j} in topic flower{i}' *
10))
{code}
!Screenshot 2021-03-13 at 00.44.43.png!
!Screenshot 2021-03-13 at 00.29.10.png!
# Make dump of tcp connections via tcpdump due ~2 sec:
{code:java}
$ tcpdump -i eth1 -w dump.pcap
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144
bytes
^C8873886 packets captured
9139050 packets received by filter
265028 packets dropped by kernel
{code}
# Load dump to Wireshark and see statistics: ~99.999% of packets is inter
broker messages, size of packets 40-160 bytes. On screen hosts with IPs
10.16.23.[157-160] is brokers:
!Screenshot 2021-03-14 at 01.46.00.png!
!Screenshot 2021-03-14 at 01.52.01.png!
*How to fix?*
# Add boolean _socket.nagle.disable_ parameter to Apache Kafka config and
provide value to kafka.network.Acceptor.accept(key) method in :
[https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/network/SocketServer.scala#L646]
# For disabled TCP_NODELAY value:
## ~400 packets/s for idle broker (instead ~12k packets/s)
## ~3k packets/s for loaded broker (instead ~150k packets/s)
!Screenshot 2021-03-16 at 21.12.17.png!
was:
h3. Problem
A large number of topic-partitions on one broker causes a burst of the host's
packets/sec metric. Most packets are small inter-broker (replication fetch)
responses of 40-160 bytes. A traffic shaper in the cloud stops coping with such
a load and causes service degradation. The original measurements from 2021 are
below.
h3. Root cause
A FetchResponse is built by {{SendBuilder}} as a {{MultiRecordsSend}}: for
every partition there is a small header buffer and the records.
{{MultiRecordsSend.writeTo()}} writes every part with its own system call, and
because {{TCP_NODELAY}} is always set, every call leaves as its own TCP
segment. With SSL it was even more, because {{SslTransportLayer}} wrapped every
buffer into its own TLS record and wrote it alone, also for the requests of the
clients.
h3. Fix
There are two changes and no new config. PR:
https://github.com/apache/kafka/pull/24170
* {{SslTransportLayer}} gives all buffers of a write to {{SSLEngine.wrap()}},
so they go into one TLS record and one write. Jun Rao proposed this in the
review.
* {{KafkaChannel.write()}} turns {{TCP_NODELAY}} off while it writes a send
that is not a {{ByteBufferSend}} (in practice a {{MultiRecordsSend}}) and turns
it on again after the write. A {{ByteBufferSend}} is written with one call, so
it is written as before. Linux sends the queued data at once when
{{TCP_NODELAY}} is turned on (tcp(7)), so the parts are joined into full
segments and nothing waits for an ACK. On other platforms this part does not
change the behaviour.
The first idea was a config to turn {{TCP_NODELAY}} off (KIP-723, PR #10333).
It was dropped: with Nagle's algorithm the end-to-end latency with acks=all is
3 times higher (p50 18 -> 57 ms on current trunk), as Colin McCabe wrote in the
KIP discussion.
h3. Results on trunk (cfcdfb7654)
3 brokers on 3 VMs of a Kubernetes cluster, 5 topics x 600 partitions, RF=3,
leaders balanced (1,000 per broker), 5 producers with acks=all and 3,000
records/s in total over all partitions. The table shows trunk and the
difference to trunk with a 95% interval:
||Listener||Packets/s on trunk||Packets/s||Bytes on the wire||Broker CPU||
|PLAINTEXT|22,330|−56%|−16%|−0.9% ± 1.3%|
|SSL|37,926|−72.5% ± 0.4%|−31.3% ± 0.3%|−0.7% ± 1.6%|
With 20 MB/s into 30 partitions the packets are already full: 5.5% fewer
packets with PLAINTEXT and 9.3% fewer with SSL, and the same broker CPU. The
details are in the notes of the PR.
h3. Links
* KIP-723:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-723%3A+Add+socket.tcp.no.delay+property+to+Kafka+Config
* dev@ discussion 2021:
https://lists.apache.org/thread/d5vwzvhyy1f7wwrxpcy173trlyx0c6vk
* dev@ discussion 2025:
https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3
* dev@ discussion 2026:
https://lists.apache.org/thread/ymvfjg6fp7k2y1pwds268vv39rq5ohyx
----
h3. Original report (2021)
*What to do?*
Add _socket.nagle.disable_ parameter to Apache Kafka config like in
[librdkafka|https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md].
*What reason of this improvement?*
A large number of topic-partitions on one broker causes burst of host's
packets/sec metric. The traffic shaper in the cloud ceases to cope with such a
load and causes service degradation.
*How to reproduce?*
# Create Kafka Cluster with 4 brokers. Amount of packet/sec is ~120.
# Add 100 topics with 100 partitions each and replication factor = 3. It is 30k
topic-partitions in total. Amount of packet/sec is ~15k.
{code:java}
import os
for i in range(100):
print(f"create topic 'flower{i}'... ", end="")
cmd = "kafka-topics.sh --create --bootstrap-server {} --topic {} --partitions
{} --replication-factor {}".format("databus.andrei-iatsuk.ec.odkl.ru:9092",
f"flower{i}", 100, 3)
code = os.system(cmd)
print("ok" if code == 0 else "error")
{code}
!Screenshot 2021-03-16 at 21.05.03.png!
# Generate server load by launching next script in 4 terminals. Amount of
packet/sec is ~130k.
{code:java}
import time
from pykafka import KafkaClient
client = KafkaClient(hosts="databus.andrei-iatsuk.ec.odkl.ru:9092")
while True:
for i in range(100):
print(f"sent message to 'flower{i}'")
with client.topics[f"flower{i}"].get_sync_producer() as producer:
for j in range(1000):
producer.produce(str.encode(f'test message {j} in topic flower{i}' *
10))
{code}
!Screenshot 2021-03-13 at 00.44.43.png!
!Screenshot 2021-03-13 at 00.29.10.png!
# Make dump of tcp connections via tcpdump due ~2 sec:
{code:java}
$ tcpdump -i eth1 -w dump.pcap
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144
bytes
^C8873886 packets captured
9139050 packets received by filter
265028 packets dropped by kernel
{code}
# Load dump to Wireshark and see statistics: ~99.999% of packets is inter
broker messages, size of packets 40-160 bytes. On screen hosts with IPs
10.16.23.[157-160] is brokers:
!Screenshot 2021-03-14 at 01.46.00.png!
!Screenshot 2021-03-14 at 01.52.01.png!
*How to fix?*
# Add boolean _socket.nagle.disable_ parameter to Apache Kafka config and
provide value to kafka.network.Acceptor.accept(key) method in :
[https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/network/SocketServer.scala#L646]
# For disabled TCP_NODELAY value:
## ~400 packets/s for idle broker (instead ~12k packets/s)
## ~3k packets/s for loaded broker (instead ~150k packets/s)
!Screenshot 2021-03-16 at 21.12.17.png!
Summary: Reduce the number of small TCP packets of a send with many
partitions (was: Coalesce small TCP segments of a send on Linux to reduce
number of packets)
> Reduce the number of small TCP packets of a send with many partitions
> ---------------------------------------------------------------------
>
> Key: KAFKA-12481
> URL: https://issues.apache.org/jira/browse/KAFKA-12481
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core, network
> Affects Versions: 0.8.2.2, 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, 2.0.1, 2.1.1,
> 2.2.2, 2.3.1, 2.4.1, 2.5.1, 2.6.1, 2.7.0, 4.4.0
> Reporter: Andrei Iatsuk
> Assignee: Andrei Iatsuk
> Priority: Major
> Attachments: Screenshot 2021-03-13 at 00.29.10.png, Screenshot
> 2021-03-13 at 00.44.43.png, Screenshot 2021-03-14 at 01.46.00.png, Screenshot
> 2021-03-14 at 01.52.01.png, Screenshot 2021-03-16 at 21.05.03.png, Screenshot
> 2021-03-16 at 21.12.17.png
>
>
> h3. Problem
> A large number of topic-partitions on one broker causes a burst of the host's
> packets/sec metric. Most packets are small inter-broker (replication fetch)
> responses of 40-160 bytes. A traffic shaper in the cloud stops coping with
> such a load and causes service degradation. The original measurements from
> 2021 are below.
> h3. Root cause
> A FetchResponse is built by {{SendBuilder}} as a {{MultiRecordsSend}}: for
> every partition there is a small header buffer and the records.
> {{MultiRecordsSend.writeTo()}} writes every part with its own system call,
> and because {{TCP_NODELAY}} is always set, every call leaves as its own TCP
> segment. With SSL it was even more, because {{SslTransportLayer}} wrapped
> every buffer into its own TLS record and wrote it alone, also for the
> requests of the clients.
> h3. Fix
> With SSL the fix is inside {{SslTransportLayer}}, without a new config. PR:
> https://github.com/apache/kafka/pull/24170
> * {{SslTransportLayer}} gives all buffers of a write to {{SSLEngine.wrap()}},
> so they go into one TLS record.
> * {{KafkaChannel.write()}} opens a write batch on the transport layer around
> the write of a send. In a batch {{SslTransportLayer}} keeps the TLS records
> in its buffer and writes them to the socket with one call when the batch ends.
> Jun Rao proposed both in the review. PLAINTEXT is not changed by this PR, see
> "Open" below.
> h3. Results on trunk (cfcdfb7654)
> 3 brokers on 3 VMs of a Kubernetes cluster, SSL. The table shows trunk and
> the difference to trunk with a 95% interval:
> ||Load||Packets/s on trunk||Packets/s||Bytes on the wire||Broker CPU||
> |5 topics x 600 partitions, RF=3, 3,000 records/s of 100 bytes,
> acks=all|37,938|−79.3% ± 0.2%|−34.4% ± 0.2%|−5.2% ± 1.5%|
> |20 MB/s into 30 partitions, RF=3, 2 consumers|88,485|−12.3% ± 0.3%|−0.6% ±
> 0.2%|−1.5% ± 1.7%|
> With one record per request (one broker, loopback) the maximum throughput
> with SSL is 10% higher (35,511 -> 39,277 records/s). With PLAINTEXT nothing
> changes. The details are in the notes of the PR.
> h3. History of the approaches
> # 2021: a config to turn {{TCP_NODELAY}} off (KIP-723, PR #10333). It was
> dropped. With Nagle's algorithm the end-to-end latency with acks=all is 3
> times higher (p50 18 -> 57 ms on current trunk), as Colin McCabe wrote in the
> KIP discussion. KIP-723 is marked as discarded.
> # 2026-11, first version of PR #24170: {{KafkaChannel.write()}} turned
> {{TCP_NODELAY}} off while it wrote a send, and on again after the write.
> Linux sends the queued data at once when {{TCP_NODELAY}} is turned on, so the
> parts left as full segments and nothing waited for an ACK. It removed 56% of
> the packets with PLAINTEXT and 73% with SSL, with the same CPU and latency.
> It was removed after the review for two reasons. It depends on the Linux TCP
> stack, and other stacks that report Linux (gVisor, kernel-bypass libraries)
> may behave differently. And it cannot be turned off without a new config,
> which needs a KIP.
> # 2026-11, current version: the fix inside {{SslTransportLayer}} described
> above. It does not depend on the kernel.
> h3. Open
> With PLAINTEXT {{MultiRecordsSend}} still writes every part with its own
> system call. An idea for a follow-up is the same write batch in user space:
> small parts are collected and written with one call, large parts still go
> with zero-copy.
> h3. Links
> * KIP-723:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-723%3A+Add+socket.tcp.no.delay+property+to+Kafka+Config
> * dev@ discussion 2021:
> https://lists.apache.org/thread/d5vwzvhyy1f7wwrxpcy173trlyx0c6vk
> * dev@ discussion 2025:
> https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3
> * dev@ discussion 2026:
> https://lists.apache.org/thread/ymvfjg6fp7k2y1pwds268vv39rq5ohyx
> ----
> h3. Original report (2021)
> *What to do?*
> Add _socket.nagle.disable_ parameter to Apache Kafka config like in
> [librdkafka|https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md].
> *What reason of this improvement?*
> A large number of topic-partitions on one broker causes burst of host's
> packets/sec metric. The traffic shaper in the cloud ceases to cope with such
> a load and causes service degradation.
> *How to reproduce?*
> # Create Kafka Cluster with 4 brokers. Amount of packet/sec is ~120.
> # Add 100 topics with 100 partitions each and replication factor = 3. It is
> 30k topic-partitions in total. Amount of packet/sec is ~15k.
> {code:java}
> import os
> for i in range(100):
> print(f"create topic 'flower{i}'... ", end="")
> cmd = "kafka-topics.sh --create --bootstrap-server {} --topic {}
> --partitions {} --replication-factor
> {}".format("databus.andrei-iatsuk.ec.odkl.ru:9092", f"flower{i}", 100, 3)
> code = os.system(cmd)
> print("ok" if code == 0 else "error")
> {code}
> !Screenshot 2021-03-16 at 21.05.03.png!
> # Generate server load by launching next script in 4 terminals. Amount of
> packet/sec is ~130k.
> {code:java}
> import time
> from pykafka import KafkaClient
> client = KafkaClient(hosts="databus.andrei-iatsuk.ec.odkl.ru:9092")
> while True:
> for i in range(100):
> print(f"sent message to 'flower{i}'")
> with client.topics[f"flower{i}"].get_sync_producer() as producer:
> for j in range(1000):
> producer.produce(str.encode(f'test message {j} in topic flower{i}' *
> 10))
> {code}
> !Screenshot 2021-03-13 at 00.44.43.png!
> !Screenshot 2021-03-13 at 00.29.10.png!
> # Make dump of tcp connections via tcpdump due ~2 sec:
> {code:java}
> $ tcpdump -i eth1 -w dump.pcap
> tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144
> bytes
> ^C8873886 packets captured
> 9139050 packets received by filter
> 265028 packets dropped by kernel
> {code}
> # Load dump to Wireshark and see statistics: ~99.999% of packets is inter
> broker messages, size of packets 40-160 bytes. On screen hosts with IPs
> 10.16.23.[157-160] is brokers:
> !Screenshot 2021-03-14 at 01.46.00.png!
> !Screenshot 2021-03-14 at 01.52.01.png!
> *How to fix?*
> # Add boolean _socket.nagle.disable_ parameter to Apache Kafka config and
> provide value to kafka.network.Acceptor.accept(key) method in :
> [https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/network/SocketServer.scala#L646]
> # For disabled TCP_NODELAY value:
> ## ~400 packets/s for idle broker (instead ~12k packets/s)
> ## ~3k packets/s for loaded broker (instead ~150k packets/s)
> !Screenshot 2021-03-16 at 21.12.17.png!
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)