Hi all,

Sorry for the very long silence on KIP-723. I came back to it, and I would like 
to answer Colin's question from 2021 and Francois' message 
(https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3).

Why there are so many small packets: a FetchResponse is built by SendBuilder as 
a MultiRecordsSend. For every partition it has a small header buffer and the 
records. MultiRecordsSend.writeTo() writes each part with its own write() or 
transferTo() call, and because TCP_NODELAY is set, each call goes out as its 
own TCP segment. So the number of packets grows with the number of partitions 
in one response. This happens with PLAINTEXT too, Francois' capture shows 
exactly this pattern (74, 670, 56, 670, ... bytes). With SSL it is even more, 
because every buffer becomes its own TLS record.

First I measured the idea of this KIP on current trunk: with TCP_NODELAY off, 
the packets go down a lot, but the end-to-end latency (acks=all) is 3 times 
higher (p50 18 -> 57 ms), even on loopback. So Colin was right, and I don't 
want to propose this config anymore.

Instead I would like to fix the small writes. In KafkaChannel.write(), 
TCP_NODELAY is turned off before send.writeTo() and turned on again after it. 
Linux documents that turning TCP_NODELAY on flushes the pending data at once 
(tcp(7)), so nothing waits for an ACK, but the parts written in one call are 
joined into full segments. It is only done on Linux; on other platforms nothing 
changes. There is no new config. PR: https://github.com/apache/kafka/pull/24170

Results on a local cluster (3 brokers on one host, 5 topics x 600 partitions, 
RF=3, 5 producers with acks=all and 3,000 records/s in total over all 
partitions, brokers and clients on the same build, baseline is trunk 
cfcdfb7654):

                      packets/s              broker CPU      e2e latency p50 / 
p99
  PLAINTEXT   18,838 -> 2,633 (-86%)    232% -> 209%    17.6 / 87 ms -> 17.0 / 
91 ms
  SSL         31,150 -> 1,386 (-96%)    231% -> 228%    17.3 / 86 ms -> 17.3 / 
111 ms

I also checked the rare 500-600 ms outliers with acks=all, which also exist on 
trunk: their number is the same with and without the change. All details are in 
the PR.

What do you think about this direction? And since there is no new config or 
other public interface, is it right that it needs no KIP, so KIP-723 can be 
discarded?

Best regards,
Andrei Iatsuk

Reply via email to