Xuze Yang created KAFKA-21527:
---------------------------------

             Summary: Improve acks=all produce throughput by releasing the 
connection once the request is processed locally
                 Key: KAFKA-21527
                 URL: https://issues.apache.org/jira/browse/KAFKA-21527
             Project: Kafka
          Issue Type: Improvement
          Components: network
    Affects Versions: 3.3.2
            Reporter: Xuze Yang


h3. Problem

A broker processes {*}only one request per connection at a time{*}. 
{{SocketServer.Processor}} mutes the channel when a request is read 
({{{}REQUEST_RECEIVED{}}}) and only unmutes it after the response has been 
written ({{{}RESPONSE_SENT{}}}). This is documented behaviour:
{quote}"The broker's request processing allows only a single in-flight request 
per connection in order to guarantee this ordering." — 
[https://kafka.apache.org/protocol]
{quote}
For {{acks=all}} produce requests this is very expensive. The request has been 
appended to the local log within a few milliseconds, but the response cannot be 
produced until every in-sync replica has fetched the records. During that 
window the whole connection is blocked, so a producer that keeps 
{{max.in.flight.requests.per.connection}} requests in flight gets no benefit 
from pipelining: the broker never reads the next request until the previous one 
has been fully acknowledged and its response written.

The same applies to any request whose response is delayed for reasons unrelated 
to the connection: {{acks=all}} produce, and (in other deployments) 
diskless/remote-storage produce.
h3. Measured impact

Environment: 48 cores / 125 GB / SSD, 1 partition with 2 replicas, 10 KB 
records, single producer, default 
{{{}max.in.flight.requests.per.connection=5{}}}.
||Scenario||Before||After||Change||
|{{{}acks=all{}}}, default in-flight|24.38 MB/s|*54.48 MB/s*|*2.23x*|
|{{{}acks=all{}}}, avg latency|811 ms|*75.56 ms*|*10.7x lower*|
|{{{}acks=all{}}}, in-flight=100|25.87 MB/s|58.86 MB/s|2.28x|
|{{acks=1}} (regression check)|59.53 MB/s|59.59 MB/s|unchanged|
|single replica {{acks=all}} (regression check)|57.92 MB/s|59.35 MB/s|unchanged|

Note the current behaviour is also visible in the in-flight numbers: raising 
{{max.in.flight}} from 5 to 100 improves throughput by only ~6% today, because 
the broker serialises the requests anyway.

The gain is proportional to the request rate, so it matters most for small 
records:
||Record size||Concurrency||Before||After||Speedup||
|10 KB|1 producer|24.4 MB/s|54.5 MB/s|2.23x|
|10 KB|4 producers|65.4 MB/s|109.6 MB/s|1.68x|
|100 KB|4 producers|188.6 MB/s|229.2 MB/s|1.22x|
|100 KB|8 producers|237.4 MB/s|262.8 MB/s|1.11x|
h3. Proposed change

Keep the ordering guarantee, but move the "waiting point" from _per request_ to 
{_}per response{_}:
 # {*}Release the channel once the request has been processed locally{*}, not 
when the response has been written. When the handler knows that the response is 
not ready yet (e.g. {{ReplicaManager}} created a {{{}DelayedProduce{}}}), it 
emits a new internal {{{}LocalProcessedResponse{}}}. The processor then unmutes 
the channel so the next pipelined request can be read.
 # *Track outstanding responses per connection in a FIFO queue keyed by 
correlation id* ({{{}SocketServer.inflightResponses{}}} becomes {{{}Map[String, 
Queue[Response]]{}}}). A response is written only when it is at the head of the 
queue, so responses are still emitted strictly in request order.
 # *Extend the channel mute state machine* with the states needed to express 
"request cached, response pending" and their interaction with throttling. A 
counter ({{{}processedDelayedReqNum{}}} / {{{}throttleReqNum{}}}) is used 
instead of a boolean so that several delayed requests and several concurrent 
throttles can coexist on one connection.

h3. Ordering and correctness

The observable guarantee is unchanged: on a single connection requests are 
still processed in the order they were sent and responses are still returned in 
that order. Two mechanisms enforce it:
 * The response queue is FIFO and a response is only written when it is at the 
head.
 * The client already validates that each response's {{correlationId}} matches 
the request at the head of its in-flight queue, and fails loudly otherwise.

Empirically, with the change applied:
 * 450k+ records produced with {{acks=all}} and {{{}max.in.flight=5{}}}, 
consumed back and compared byte-for-byte: {*}no loss, no duplication, strict 
ordering{*}.
 * Unclean {{SIGKILL}} of the leader after a successful {{acks=all}} write: 
*zero data loss* (i.e. the change does not acknowledge earlier than before).
 * Zero {{Invalid correlation id}} / out-of-order errors across all client logs.
 * Zero invalid mute-state transitions in the broker logs.

h3. Compatibility
 * No wire protocol change: no new request/response fields, no API version 
bump. Clients that ship their own {{kafka-clients}} are unaffected.
 * {{acks=0}} and {{acks=1}} paths are untouched (no delayed response is 
created for them).
 * The change can be gated behind a broker config (e.g. 
{{{}socket.request.pipeline.after.local.process=true{}}}, default 
{{{}false{}}}) so that it can be rolled out gradually and disabled if needed.

h3. Risks and open questions
 # *Memory / back-pressure.* Reading further requests while responses are 
pending increases the number of in-flight requests per connection. It is 
bounded by the existing {{queued.max.requests}} and by the client's 
{{{}max.in.flight{}}}, but the interaction should be reviewed explicitly.
 # *Quota semantics.* Throttling already mutes the channel; the new "request 
cached" states have to compose with 
{{{}THROTTLE_STARTED{}}}/{{{}THROTTLE_ENDED{}}}. This is the main source of 
complexity in the implementation (9 mute states) and is the part most likely to 
need re-design during review. A first iteration could restrict the optimisation 
to the non-throttled case.
 # *Fairness across connections.* A connection that keeps pipelining more 
requests could occupy a larger share of the request queue than before. Worth 
measuring with many concurrent producers.
 # *Interaction with {{DelayedFetch}} / purgatory.* Only produce is covered by 
the prototype; fetch and other delayed responses may or may not benefit.

h3. Prior art

AutoMQ's LIP-162 (diskless storage) identifies the same head-of-line blocking 
and moves the waiting point from "per request" to "per flush cycle" for the 
same reason.
h3. Willingness to contribute

We have a working implementation against 3.3 (approximately 700 lines across 
{{{}KafkaChannel{}}}, {{{}Selector{}}}, {{{}SocketServer{}}}, 
{{{}ReplicaManager{}}}, {{{}KafkaApis{}}}) plus the test suite and benchmark 
harness described above, and are willing to submit it as a PR and iterate on 
review feedback.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to