Xuze Yang created KAFKA-21527:
---------------------------------
Summary: Improve acks=all produce throughput by releasing the
connection once the request is processed locally
Key: KAFKA-21527
URL: https://issues.apache.org/jira/browse/KAFKA-21527
Project: Kafka
Issue Type: Improvement
Components: network
Affects Versions: 3.3.2
Reporter: Xuze Yang
h3. Problem
A broker processes {*}only one request per connection at a time{*}.
{{SocketServer.Processor}} mutes the channel when a request is read
({{{}REQUEST_RECEIVED{}}}) and only unmutes it after the response has been
written ({{{}RESPONSE_SENT{}}}). This is documented behaviour:
{quote}"The broker's request processing allows only a single in-flight request
per connection in order to guarantee this ordering." —
[https://kafka.apache.org/protocol]
{quote}
For {{acks=all}} produce requests this is very expensive. The request has been
appended to the local log within a few milliseconds, but the response cannot be
produced until every in-sync replica has fetched the records. During that
window the whole connection is blocked, so a producer that keeps
{{max.in.flight.requests.per.connection}} requests in flight gets no benefit
from pipelining: the broker never reads the next request until the previous one
has been fully acknowledged and its response written.
The same applies to any request whose response is delayed for reasons unrelated
to the connection: {{acks=all}} produce, and (in other deployments)
diskless/remote-storage produce.
h3. Measured impact
Environment: 48 cores / 125 GB / SSD, 1 partition with 2 replicas, 10 KB
records, single producer, default
{{{}max.in.flight.requests.per.connection=5{}}}.
||Scenario||Before||After||Change||
|{{{}acks=all{}}}, default in-flight|24.38 MB/s|*54.48 MB/s*|*2.23x*|
|{{{}acks=all{}}}, avg latency|811 ms|*75.56 ms*|*10.7x lower*|
|{{{}acks=all{}}}, in-flight=100|25.87 MB/s|58.86 MB/s|2.28x|
|{{acks=1}} (regression check)|59.53 MB/s|59.59 MB/s|unchanged|
|single replica {{acks=all}} (regression check)|57.92 MB/s|59.35 MB/s|unchanged|
Note the current behaviour is also visible in the in-flight numbers: raising
{{max.in.flight}} from 5 to 100 improves throughput by only ~6% today, because
the broker serialises the requests anyway.
The gain is proportional to the request rate, so it matters most for small
records:
||Record size||Concurrency||Before||After||Speedup||
|10 KB|1 producer|24.4 MB/s|54.5 MB/s|2.23x|
|10 KB|4 producers|65.4 MB/s|109.6 MB/s|1.68x|
|100 KB|4 producers|188.6 MB/s|229.2 MB/s|1.22x|
|100 KB|8 producers|237.4 MB/s|262.8 MB/s|1.11x|
h3. Proposed change
Keep the ordering guarantee, but move the "waiting point" from _per request_ to
{_}per response{_}:
# {*}Release the channel once the request has been processed locally{*}, not
when the response has been written. When the handler knows that the response is
not ready yet (e.g. {{ReplicaManager}} created a {{{}DelayedProduce{}}}), it
emits a new internal {{{}LocalProcessedResponse{}}}. The processor then unmutes
the channel so the next pipelined request can be read.
# *Track outstanding responses per connection in a FIFO queue keyed by
correlation id* ({{{}SocketServer.inflightResponses{}}} becomes {{{}Map[String,
Queue[Response]]{}}}). A response is written only when it is at the head of the
queue, so responses are still emitted strictly in request order.
# *Extend the channel mute state machine* with the states needed to express
"request cached, response pending" and their interaction with throttling. A
counter ({{{}processedDelayedReqNum{}}} / {{{}throttleReqNum{}}}) is used
instead of a boolean so that several delayed requests and several concurrent
throttles can coexist on one connection.
h3. Ordering and correctness
The observable guarantee is unchanged: on a single connection requests are
still processed in the order they were sent and responses are still returned in
that order. Two mechanisms enforce it:
* The response queue is FIFO and a response is only written when it is at the
head.
* The client already validates that each response's {{correlationId}} matches
the request at the head of its in-flight queue, and fails loudly otherwise.
Empirically, with the change applied:
* 450k+ records produced with {{acks=all}} and {{{}max.in.flight=5{}}},
consumed back and compared byte-for-byte: {*}no loss, no duplication, strict
ordering{*}.
* Unclean {{SIGKILL}} of the leader after a successful {{acks=all}} write:
*zero data loss* (i.e. the change does not acknowledge earlier than before).
* Zero {{Invalid correlation id}} / out-of-order errors across all client logs.
* Zero invalid mute-state transitions in the broker logs.
h3. Compatibility
* No wire protocol change: no new request/response fields, no API version
bump. Clients that ship their own {{kafka-clients}} are unaffected.
* {{acks=0}} and {{acks=1}} paths are untouched (no delayed response is
created for them).
* The change can be gated behind a broker config (e.g.
{{{}socket.request.pipeline.after.local.process=true{}}}, default
{{{}false{}}}) so that it can be rolled out gradually and disabled if needed.
h3. Risks and open questions
# *Memory / back-pressure.* Reading further requests while responses are
pending increases the number of in-flight requests per connection. It is
bounded by the existing {{queued.max.requests}} and by the client's
{{{}max.in.flight{}}}, but the interaction should be reviewed explicitly.
# *Quota semantics.* Throttling already mutes the channel; the new "request
cached" states have to compose with
{{{}THROTTLE_STARTED{}}}/{{{}THROTTLE_ENDED{}}}. This is the main source of
complexity in the implementation (9 mute states) and is the part most likely to
need re-design during review. A first iteration could restrict the optimisation
to the non-throttled case.
# *Fairness across connections.* A connection that keeps pipelining more
requests could occupy a larger share of the request queue than before. Worth
measuring with many concurrent producers.
# *Interaction with {{DelayedFetch}} / purgatory.* Only produce is covered by
the prototype; fetch and other delayed responses may or may not benefit.
h3. Prior art
AutoMQ's LIP-162 (diskless storage) identifies the same head-of-line blocking
and moves the waiting point from "per request" to "per flush cycle" for the
same reason.
h3. Willingness to contribute
We have a working implementation against 3.3 (approximately 700 lines across
{{{}KafkaChannel{}}}, {{{}Selector{}}}, {{{}SocketServer{}}},
{{{}ReplicaManager{}}}, {{{}KafkaApis{}}}) plus the test suite and benchmark
harness described above, and are willing to submit it as a PR and iterate on
review feedback.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)