[ 
https://issues.apache.org/jira/browse/KAFKA-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrei Iatsuk updated KAFKA-12481:
----------------------------------
    Description: 
h3. Problem
A large number of topic-partitions on one broker causes a burst of the host's 
packets/sec metric. Most packets are small inter-broker (replication fetch) 
responses of 40-160 bytes. A traffic shaper in the cloud stops coping with such 
a load and causes service degradation. The original measurements from 2021 are 
below.

h3. Root cause
A FetchResponse is built by {{SendBuilder}} as a {{MultiRecordsSend}}: for 
every partition there is a small header buffer and the records. 
{{MultiRecordsSend.writeTo()}} writes every part with its own system call, and 
because {{TCP_NODELAY}} is always set, every call leaves as its own TCP 
segment. With SSL it was even more, because {{SslTransportLayer}} wrapped every 
buffer into its own TLS record and wrote it alone, also for the requests of the 
clients.

h3. Fix
With SSL the fix is inside {{SslTransportLayer}}, without a new config. PR: 
https://github.com/apache/kafka/pull/24170
* {{SslTransportLayer}} gives all buffers of a write to {{SSLEngine.wrap()}}, 
so they go into one TLS record.
* {{KafkaChannel.write()}} opens a write batch on the transport layer around 
the write of a send. In a batch {{SslTransportLayer}} keeps the TLS records in 
its buffer and writes them to the socket with one call when the batch ends.

Jun Rao proposed both in the review. PLAINTEXT is not changed by this PR, see 
"Open" below.

h3. Results on trunk (cfcdfb7654)
3 brokers on 3 VMs of a Kubernetes cluster, SSL. The table shows trunk and the 
difference to trunk with a 95% interval:
||Load||Packets/s on trunk||Packets/s||Bytes on the wire||Broker CPU||
|5 topics x 600 partitions, RF=3, 3,000 records/s of 100 bytes, 
acks=all|37,938|−79.3% ± 0.2%|−34.4% ± 0.2%|−5.2% ± 1.5%|
|20 MB/s into 30 partitions, RF=3, 2 consumers|88,485|−12.3% ± 0.3%|−0.6% ± 
0.2%|−1.5% ± 1.7%|

With one record per request (one broker, loopback) the maximum throughput with 
SSL is 10% higher (35,511 -> 39,277 records/s). With PLAINTEXT nothing changes. 
The details are in the notes of the PR.

h3. History of the approaches
# 2021: a config to turn {{TCP_NODELAY}} off (KIP-723, PR #10333). It was 
dropped. With Nagle's algorithm the end-to-end latency with acks=all is 3 times 
higher (p50 18 -> 57 ms on current trunk), as Colin McCabe wrote in the KIP 
discussion. KIP-723 is marked as discarded.
# 2026-11, first version of PR #24170: {{KafkaChannel.write()}} turned 
{{TCP_NODELAY}} off while it wrote a send, and on again after the write. Linux 
sends the queued data at once when {{TCP_NODELAY}} is turned on, so the parts 
left as full segments and nothing waited for an ACK. It removed 56% of the 
packets with PLAINTEXT and 73% with SSL, with the same CPU and latency. It was 
removed after the review for two reasons. It depends on the Linux TCP stack, 
and other stacks that report Linux (gVisor, kernel-bypass libraries) may behave 
differently. And it cannot be turned off without a new config, which needs a 
KIP.
# 2026-11, current version: the fix inside {{SslTransportLayer}} described 
above. It does not depend on the kernel.

h3. Open
With PLAINTEXT {{MultiRecordsSend}} still writes every part with its own system 
call. An idea for a follow-up is the same write batch in user space: small 
parts are collected and written with one call, large parts still go with 
zero-copy.

h3. Links
* KIP-723: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-723%3A+Add+socket.tcp.no.delay+property+to+Kafka+Config
* dev@ discussion 2021: 
https://lists.apache.org/thread/d5vwzvhyy1f7wwrxpcy173trlyx0c6vk
* dev@ discussion 2025: 
https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3
* dev@ discussion 2026: 
https://lists.apache.org/thread/ymvfjg6fp7k2y1pwds268vv39rq5ohyx

----
h3. Original report (2021)
*What to do?*

Add _socket.nagle.disable_ parameter to Apache Kafka config like in 
[librdkafka|https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md].

*What reason of this improvement?*

A large number of topic-partitions on one broker causes burst of host's 
packets/sec metric. The traffic shaper in the cloud ceases to cope with such a 
load and causes service degradation.

*How to reproduce?*
# Create Kafka Cluster with 4 brokers. Amount of packet/sec is ~120.
# Add 100 topics with 100 partitions each and replication factor = 3. It is 30k 
topic-partitions in total. Amount of packet/sec is ~15k.
{code:java}
import os
for i in range(100):
  print(f"create topic 'flower{i}'... ", end="")
  cmd = "kafka-topics.sh --create --bootstrap-server {} --topic {} --partitions 
{} --replication-factor {}".format("databus.andrei-iatsuk.ec.odkl.ru:9092", 
f"flower{i}", 100, 3)
  code = os.system(cmd)
  print("ok" if code == 0 else "error")
{code}
!Screenshot 2021-03-16 at 21.05.03.png!
# Generate server load by launching next script in 4 terminals. Amount of 
packet/sec is ~130k.
{code:java}
import time
from pykafka import KafkaClient
client = KafkaClient(hosts="databus.andrei-iatsuk.ec.odkl.ru:9092")
while True:
  for i in range(100):
    print(f"sent message to 'flower{i}'")
    with client.topics[f"flower{i}"].get_sync_producer() as producer:
      for j in range(1000):
        producer.produce(str.encode(f'test message {j} in topic flower{i}' * 
10))
{code}
!Screenshot 2021-03-13 at 00.44.43.png! 
 !Screenshot 2021-03-13 at 00.29.10.png!
# Make dump of tcp connections via tcpdump due ~2 sec:
{code:java}
$ tcpdump -i eth1 -w dump.pcap
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 
bytes
^C8873886 packets captured
9139050 packets received by filter
265028 packets dropped by kernel
{code}
# Load dump to Wireshark and see statistics: ~99.999% of packets is inter 
broker messages, size of packets 40-160 bytes. On screen hosts with IPs 
10.16.23.[157-160] is brokers:
 !Screenshot 2021-03-14 at 01.46.00.png! 
 !Screenshot 2021-03-14 at 01.52.01.png!

*How to fix?*
# Add boolean _socket.nagle.disable_ parameter to Apache Kafka config and 
provide value to kafka.network.Acceptor.accept(key) method in : 
[https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/network/SocketServer.scala#L646]
# For disabled TCP_NODELAY value:
## ~400 packets/s for idle broker (instead ~12k packets/s)
## ~3k packets/s for loaded broker (instead ~150k packets/s)
 !Screenshot 2021-03-16 at 21.12.17.png!

  was:
h3. Problem
A large number of topic-partitions on one broker causes a burst of the host's 
packets/sec metric. Most packets are small inter-broker (replication fetch) 
responses of 40-160 bytes. A traffic shaper in the cloud stops coping with such 
a load and causes service degradation. The original measurements from 2021 are 
below.

h3. Root cause
A FetchResponse is built by {{SendBuilder}} as a {{MultiRecordsSend}}: for 
every partition there is a small header buffer and the records. 
{{MultiRecordsSend.writeTo()}} writes every part with its own system call, and 
because {{TCP_NODELAY}} is always set, every call leaves as its own TCP 
segment. With SSL it was even more, because {{SslTransportLayer}} wrapped every 
buffer into its own TLS record and wrote it alone, also for the requests of the 
clients.

h3. Fix
There are two changes and no new config. PR: 
https://github.com/apache/kafka/pull/24170
* {{SslTransportLayer}} gives all buffers of a write to {{SSLEngine.wrap()}}, 
so they go into one TLS record and one write. Jun Rao proposed this in the 
review.
* {{KafkaChannel.write()}} turns {{TCP_NODELAY}} off while it writes a send 
that is not a {{ByteBufferSend}} (in practice a {{MultiRecordsSend}}) and turns 
it on again after the write. A {{ByteBufferSend}} is written with one call, so 
it is written as before. Linux sends the queued data at once when 
{{TCP_NODELAY}} is turned on (tcp(7)), so the parts are joined into full 
segments and nothing waits for an ACK. On other platforms this part does not 
change the behaviour.

The first idea was a config to turn {{TCP_NODELAY}} off (KIP-723, PR #10333). 
It was dropped: with Nagle's algorithm the end-to-end latency with acks=all is 
3 times higher (p50 18 -> 57 ms on current trunk), as Colin McCabe wrote in the 
KIP discussion.

h3. Results on trunk (cfcdfb7654)
3 brokers on 3 VMs of a Kubernetes cluster, 5 topics x 600 partitions, RF=3, 
leaders balanced (1,000 per broker), 5 producers with acks=all and 3,000 
records/s in total over all partitions. The table shows trunk and the 
difference to trunk with a 95% interval:
||Listener||Packets/s on trunk||Packets/s||Bytes on the wire||Broker CPU||
|PLAINTEXT|22,330|−56%|−16%|−0.9% ± 1.3%|
|SSL|37,926|−72.5% ± 0.4%|−31.3% ± 0.3%|−0.7% ± 1.6%|

With 20 MB/s into 30 partitions the packets are already full: 5.5% fewer 
packets with PLAINTEXT and 9.3% fewer with SSL, and the same broker CPU. The 
details are in the notes of the PR.

h3. Links
* KIP-723: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-723%3A+Add+socket.tcp.no.delay+property+to+Kafka+Config
* dev@ discussion 2021: 
https://lists.apache.org/thread/d5vwzvhyy1f7wwrxpcy173trlyx0c6vk
* dev@ discussion 2025: 
https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3
* dev@ discussion 2026: 
https://lists.apache.org/thread/ymvfjg6fp7k2y1pwds268vv39rq5ohyx

----
h3. Original report (2021)
*What to do?*

Add _socket.nagle.disable_ parameter to Apache Kafka config like in 
[librdkafka|https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md].

*What reason of this improvement?*

A large number of topic-partitions on one broker causes burst of host's 
packets/sec metric. The traffic shaper in the cloud ceases to cope with such a 
load and causes service degradation.

*How to reproduce?*
# Create Kafka Cluster with 4 brokers. Amount of packet/sec is ~120.
# Add 100 topics with 100 partitions each and replication factor = 3. It is 30k 
topic-partitions in total. Amount of packet/sec is ~15k.
{code:java}
import os
for i in range(100):
  print(f"create topic 'flower{i}'... ", end="")
  cmd = "kafka-topics.sh --create --bootstrap-server {} --topic {} --partitions 
{} --replication-factor {}".format("databus.andrei-iatsuk.ec.odkl.ru:9092", 
f"flower{i}", 100, 3)
  code = os.system(cmd)
  print("ok" if code == 0 else "error")
{code}
!Screenshot 2021-03-16 at 21.05.03.png!
# Generate server load by launching next script in 4 terminals. Amount of 
packet/sec is ~130k.
{code:java}
import time
from pykafka import KafkaClient
client = KafkaClient(hosts="databus.andrei-iatsuk.ec.odkl.ru:9092")
while True:
  for i in range(100):
    print(f"sent message to 'flower{i}'")
    with client.topics[f"flower{i}"].get_sync_producer() as producer:
      for j in range(1000):
        producer.produce(str.encode(f'test message {j} in topic flower{i}' * 
10))
{code}
!Screenshot 2021-03-13 at 00.44.43.png! 
 !Screenshot 2021-03-13 at 00.29.10.png!
# Make dump of tcp connections via tcpdump due ~2 sec:
{code:java}
$ tcpdump -i eth1 -w dump.pcap
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 
bytes
^C8873886 packets captured
9139050 packets received by filter
265028 packets dropped by kernel
{code}
# Load dump to Wireshark and see statistics: ~99.999% of packets is inter 
broker messages, size of packets 40-160 bytes. On screen hosts with IPs 
10.16.23.[157-160] is brokers:
 !Screenshot 2021-03-14 at 01.46.00.png! 
 !Screenshot 2021-03-14 at 01.52.01.png!

*How to fix?*
# Add boolean _socket.nagle.disable_ parameter to Apache Kafka config and 
provide value to kafka.network.Acceptor.accept(key) method in : 
[https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/network/SocketServer.scala#L646]
# For disabled TCP_NODELAY value:
## ~400 packets/s for idle broker (instead ~12k packets/s)
## ~3k packets/s for loaded broker (instead ~150k packets/s)
 !Screenshot 2021-03-16 at 21.12.17.png!

        Summary: Reduce the number of small TCP packets of a send with many 
partitions  (was: Coalesce small TCP segments of a send on Linux to reduce 
number of packets)

> Reduce the number of small TCP packets of a send with many partitions
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-12481
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12481
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, core, network
>    Affects Versions: 0.8.2.2, 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, 2.0.1, 2.1.1, 
> 2.2.2, 2.3.1, 2.4.1, 2.5.1, 2.6.1, 2.7.0, 4.4.0
>            Reporter: Andrei Iatsuk
>            Assignee: Andrei Iatsuk
>            Priority: Major
>         Attachments: Screenshot 2021-03-13 at 00.29.10.png, Screenshot 
> 2021-03-13 at 00.44.43.png, Screenshot 2021-03-14 at 01.46.00.png, Screenshot 
> 2021-03-14 at 01.52.01.png, Screenshot 2021-03-16 at 21.05.03.png, Screenshot 
> 2021-03-16 at 21.12.17.png
>
>
> h3. Problem
> A large number of topic-partitions on one broker causes a burst of the host's 
> packets/sec metric. Most packets are small inter-broker (replication fetch) 
> responses of 40-160 bytes. A traffic shaper in the cloud stops coping with 
> such a load and causes service degradation. The original measurements from 
> 2021 are below.
> h3. Root cause
> A FetchResponse is built by {{SendBuilder}} as a {{MultiRecordsSend}}: for 
> every partition there is a small header buffer and the records. 
> {{MultiRecordsSend.writeTo()}} writes every part with its own system call, 
> and because {{TCP_NODELAY}} is always set, every call leaves as its own TCP 
> segment. With SSL it was even more, because {{SslTransportLayer}} wrapped 
> every buffer into its own TLS record and wrote it alone, also for the 
> requests of the clients.
> h3. Fix
> With SSL the fix is inside {{SslTransportLayer}}, without a new config. PR: 
> https://github.com/apache/kafka/pull/24170
> * {{SslTransportLayer}} gives all buffers of a write to {{SSLEngine.wrap()}}, 
> so they go into one TLS record.
> * {{KafkaChannel.write()}} opens a write batch on the transport layer around 
> the write of a send. In a batch {{SslTransportLayer}} keeps the TLS records 
> in its buffer and writes them to the socket with one call when the batch ends.
> Jun Rao proposed both in the review. PLAINTEXT is not changed by this PR, see 
> "Open" below.
> h3. Results on trunk (cfcdfb7654)
> 3 brokers on 3 VMs of a Kubernetes cluster, SSL. The table shows trunk and 
> the difference to trunk with a 95% interval:
> ||Load||Packets/s on trunk||Packets/s||Bytes on the wire||Broker CPU||
> |5 topics x 600 partitions, RF=3, 3,000 records/s of 100 bytes, 
> acks=all|37,938|−79.3% ± 0.2%|−34.4% ± 0.2%|−5.2% ± 1.5%|
> |20 MB/s into 30 partitions, RF=3, 2 consumers|88,485|−12.3% ± 0.3%|−0.6% ± 
> 0.2%|−1.5% ± 1.7%|
> With one record per request (one broker, loopback) the maximum throughput 
> with SSL is 10% higher (35,511 -> 39,277 records/s). With PLAINTEXT nothing 
> changes. The details are in the notes of the PR.
> h3. History of the approaches
> # 2021: a config to turn {{TCP_NODELAY}} off (KIP-723, PR #10333). It was 
> dropped. With Nagle's algorithm the end-to-end latency with acks=all is 3 
> times higher (p50 18 -> 57 ms on current trunk), as Colin McCabe wrote in the 
> KIP discussion. KIP-723 is marked as discarded.
> # 2026-11, first version of PR #24170: {{KafkaChannel.write()}} turned 
> {{TCP_NODELAY}} off while it wrote a send, and on again after the write. 
> Linux sends the queued data at once when {{TCP_NODELAY}} is turned on, so the 
> parts left as full segments and nothing waited for an ACK. It removed 56% of 
> the packets with PLAINTEXT and 73% with SSL, with the same CPU and latency. 
> It was removed after the review for two reasons. It depends on the Linux TCP 
> stack, and other stacks that report Linux (gVisor, kernel-bypass libraries) 
> may behave differently. And it cannot be turned off without a new config, 
> which needs a KIP.
> # 2026-11, current version: the fix inside {{SslTransportLayer}} described 
> above. It does not depend on the kernel.
> h3. Open
> With PLAINTEXT {{MultiRecordsSend}} still writes every part with its own 
> system call. An idea for a follow-up is the same write batch in user space: 
> small parts are collected and written with one call, large parts still go 
> with zero-copy.
> h3. Links
> * KIP-723: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-723%3A+Add+socket.tcp.no.delay+property+to+Kafka+Config
> * dev@ discussion 2021: 
> https://lists.apache.org/thread/d5vwzvhyy1f7wwrxpcy173trlyx0c6vk
> * dev@ discussion 2025: 
> https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3
> * dev@ discussion 2026: 
> https://lists.apache.org/thread/ymvfjg6fp7k2y1pwds268vv39rq5ohyx
> ----
> h3. Original report (2021)
> *What to do?*
> Add _socket.nagle.disable_ parameter to Apache Kafka config like in 
> [librdkafka|https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md].
> *What reason of this improvement?*
> A large number of topic-partitions on one broker causes burst of host's 
> packets/sec metric. The traffic shaper in the cloud ceases to cope with such 
> a load and causes service degradation.
> *How to reproduce?*
> # Create Kafka Cluster with 4 brokers. Amount of packet/sec is ~120.
> # Add 100 topics with 100 partitions each and replication factor = 3. It is 
> 30k topic-partitions in total. Amount of packet/sec is ~15k.
> {code:java}
> import os
> for i in range(100):
>   print(f"create topic 'flower{i}'... ", end="")
>   cmd = "kafka-topics.sh --create --bootstrap-server {} --topic {} 
> --partitions {} --replication-factor 
> {}".format("databus.andrei-iatsuk.ec.odkl.ru:9092", f"flower{i}", 100, 3)
>   code = os.system(cmd)
>   print("ok" if code == 0 else "error")
> {code}
> !Screenshot 2021-03-16 at 21.05.03.png!
> # Generate server load by launching next script in 4 terminals. Amount of 
> packet/sec is ~130k.
> {code:java}
> import time
> from pykafka import KafkaClient
> client = KafkaClient(hosts="databus.andrei-iatsuk.ec.odkl.ru:9092")
> while True:
>   for i in range(100):
>     print(f"sent message to 'flower{i}'")
>     with client.topics[f"flower{i}"].get_sync_producer() as producer:
>       for j in range(1000):
>         producer.produce(str.encode(f'test message {j} in topic flower{i}' * 
> 10))
> {code}
> !Screenshot 2021-03-13 at 00.44.43.png! 
>  !Screenshot 2021-03-13 at 00.29.10.png!
> # Make dump of tcp connections via tcpdump due ~2 sec:
> {code:java}
> $ tcpdump -i eth1 -w dump.pcap
> tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 
> bytes
> ^C8873886 packets captured
> 9139050 packets received by filter
> 265028 packets dropped by kernel
> {code}
> # Load dump to Wireshark and see statistics: ~99.999% of packets is inter 
> broker messages, size of packets 40-160 bytes. On screen hosts with IPs 
> 10.16.23.[157-160] is brokers:
>  !Screenshot 2021-03-14 at 01.46.00.png! 
>  !Screenshot 2021-03-14 at 01.52.01.png!
> *How to fix?*
> # Add boolean _socket.nagle.disable_ parameter to Apache Kafka config and 
> provide value to kafka.network.Acceptor.accept(key) method in : 
> [https://github.com/apache/kafka/blob/2.4/core/src/main/scala/kafka/network/SocketServer.scala#L646]
> # For disabled TCP_NODELAY value:
> ## ~400 packets/s for idle broker (instead ~12k packets/s)
> ## ~3k packets/s for loaded broker (instead ~150k packets/s)
>  !Screenshot 2021-03-16 at 21.12.17.png!



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to