qianye1001 opened a new issue, #11489: URL: https://github.com/apache/rocketmq/issues/11489
### Before Creating the Enhancement Request - [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature. ### Summary With `popConsumerKVServiceEnable=true`, ACK and `ChangeInvisibleTime` can spend substantial time waiting inside synchronous RocksDB writes. Both request types share the Broker ACK executor, so bursts of invisible-time changes can occupy its workers and delay otherwise unrelated ACK requests. Improve ACK/CHANGE throughput and tail latency by batching POP KV mutations across requests and completing responses asynchronously after persistence, while keeping synchronous WAL durability. ### Motivation The current POP KV path has the following behavior: - An ACK that misses the POP checkpoint cache calls `PopConsumerRocksdbStore.deleteRecords()` directly. Although the service method is named `ackAsync()`, the RocksDB write happens before the method returns. - `changeInvisibilityDuration()` writes the replacement checkpoint synchronously and, when necessary, deletes the old checkpoint in another synchronous write. Even when the old checkpoint is cached, the replacement checkpoint still requires a storage write. - Both write paths use `WriteOptions.sync=true` and WAL enabled. The request worker remains occupied until `db.write()` returns; WAL synchronization and RocksDB write stalls can therefore consume the shared ACK executor's capacity. - POP checkpoint writes, cache flushing and expired-checkpoint/revive maintenance also write to the same database. Bursty renewals, client-disconnection cleanup or other batches of invisible-time changes can consequently increase ACK queueing latency and reduce consumption throughput. Increasing the request thread count does not remove the synchronous storage work or aggregate mutations at the Broker layer. RocksDB can already group concurrent writers internally, but callers still wait in its synchronous write API. A workload for evaluating this problem is a small ACK executor serving a steady ACK stream together with periodic CHANGE bursts. Use `enablePopBufferMerge=false` to exercise storage ACKs directly, and repeat with the cache enabled to cover renewal writes and cache misses. Measure ACK/CHANGE latency, executor queueing, completed requests and WAL synchronization activity. This issue does not claim a measured end-to-end throughput improvement. ### Describe the Solution You'd Like - Introduce a shared, bounded POP KV mutation queue and one batch writer. Drain currently available requests into a `WriteBatch` without deliberately delaying low-load requests to collect a batch. - Return real persistence Futures to ACK and CHANGE callers. Send success only after the required write completes; propagate write failures and queue rejection to the response. - Make the POP KV CHANGE response asynchronous independently of the legacy `appendCkAsync` / `appendAckAsync` settings. - Route checkpoint creation and background checkpoint maintenance through the same writer. Preserve operation order, cached-checkpoint safety, same-key replacement behavior and scan-cursor advancement after successful persistence. - Preserve `sync=true` and WAL enabled, bound outstanding work, and drain accepted writes before closing the database. ### Describe Alternatives You've Considered - **More ACK workers:** allows more synchronous callers to wait, but does not address request-thread occupancy or Broker-level batching. - **One background write per request:** releases request workers, but serializing individual small writes without batching can reduce throughput. - **`sync=false`:** changes the durability guarantee and still does not provide a callback-based nonblocking write API. - **An ACK worker elected as batch leader:** can bound blocked ACK workers to one; a dedicated writer instead isolates the request pool from storage stalls and also serves background producers. ### Additional Context Inspected on `develop` at `8ab8191478115d5e2f5344127fba704267fdb659` (project version 5.5.1, RocksDB JNI 8.4.4). Related: #10489 proposes a new batch CHANGE protocol across Proxy/Broker. This issue concerns Broker-side aggregation and asynchronous persistence for existing ACK/CHANGE requests, without requiring a new client or wire protocol. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
