Hi all, Sorry for the very long silence on KIP-723. I came back to it, and I would like to answer Colin's question from 2021 and Francois' message (https://lists.apache.org/thread/z1n5vz8fd4to15rj1n0p9wt5rvlzr6j3).
Why there are so many small packets: a FetchResponse is built by SendBuilder as a MultiRecordsSend. For every partition it has a small header buffer and the records. MultiRecordsSend.writeTo() writes each part with its own write() or transferTo() call, and because TCP_NODELAY is set, each call goes out as its own TCP segment. So the number of packets grows with the number of partitions in one response. This happens with PLAINTEXT too, Francois' capture shows exactly this pattern (74, 670, 56, 670, ... bytes). With SSL it is even more, because every buffer becomes its own TLS record. First I measured the idea of this KIP on current trunk: with TCP_NODELAY off, the packets go down a lot, but the end-to-end latency (acks=all) is 3 times higher (p50 18 -> 57 ms), even on loopback. So Colin was right, and I don't want to propose this config anymore. Instead I would like to fix the small writes. In KafkaChannel.write(), TCP_NODELAY is turned off before send.writeTo() and turned on again after it. Linux documents that turning TCP_NODELAY on flushes the pending data at once (tcp(7)), so nothing waits for an ACK, but the parts written in one call are joined into full segments. It is only done on Linux; on other platforms nothing changes. There is no new config. PR: https://github.com/apache/kafka/pull/24170 Results on a local cluster (3 brokers on one host, 5 topics x 600 partitions, RF=3, 5 producers with acks=all and 3,000 records/s in total over all partitions, brokers and clients on the same build, baseline is trunk cfcdfb7654): packets/s broker CPU e2e latency p50 / p99 PLAINTEXT 18,838 -> 2,633 (-86%) 232% -> 209% 17.6 / 87 ms -> 17.0 / 91 ms SSL 31,150 -> 1,386 (-96%) 231% -> 228% 17.3 / 86 ms -> 17.3 / 111 ms I also checked the rare 500-600 ms outliers with acks=all, which also exist on trunk: their number is the same with and without the change. All details are in the PR. What do you think about this direction? And since there is no new config or other public interface, is it right that it needs no KIP, so KIP-723 can be discarded? Best regards, Andrei Iatsuk
