platinumhamburg opened a new issue, #3454: URL: https://github.com/apache/fluss/issues/3454
### Search before asking - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar. ### Motivation Primary key table read/write operations involving RocksDB are currently executed on Netty worker threads while holding the bucket lock. When RocksDB slows down — due to disk I/O bottlenecks, compaction pressure, or compute saturation — the bucket lock is held for extended periods. This causes a cascading failure pattern: 1. RocksDB latency increases. 2. Netty worker threads block on the bucket lock waiting for the slow RocksDB operation to complete. 3. RPC requests of **all types** (not just KV writes) pile up in the queue because worker threads are exhausted. 4. The cluster enters a severe "stalling oscillation" — request timeouts cascade, healthy buckets become unreachable, and overall cluster health degrades far beyond the scope of the original slowdown. The core issue: a single slow bucket's RocksDB degradation propagates into a cluster-wide availability problem. ### Solution Introduce a backpressure mechanism that decouples RocksDB latency from Netty worker thread occupancy: 1. **Async RocksDB execution**: Move RocksDB operations off Netty worker threads into a dedicated thread pool, so worker threads are never blocked by slow I/O. 2. **Per-bucket backpressure signaling**: When a bucket's RocksDB operation queue depth or latency exceeds a threshold, apply backpressure to incoming write requests for that bucket (e.g., reject with `RETRIABLE_BACKPRESSURE` or throttle at the request handler level). 3. **Lock scoping**: Ensure the bucket lock is not held across the async boundary, or narrow the critical section so that lock contention does not amplify the slowdown. 4. **Isolation guarantee**: Reads and operations on other buckets must not be impacted by one bucket's RocksDB degradation. ### Anything else? _No response_ ### Willingness to contribute - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
