platinumhamburg opened a new issue, #2177: URL: https://github.com/apache/fluss/issues/2177
### Search before asking - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar. ### Motivation RocksDB's flush and compaction operations can generate significant I/O pressure, potentially impacting: - **System stability**: Uncontrolled I/O may saturate disk bandwidth, affecting other operations - **Query latency**: High I/O contention can degrade read/write performance - **Resource predictability**: Difficult to manage multi-tenant TabletServer resource allocation RocksDB provides a built-in rate limiter mechanism to control flush and compaction write rates, but Fluss currently doesn't expose this capability to users. ### Solution ### 1. Shared Rate Limiter Architecture - **Server-level rate limiting**: A single rate limiter shared across all RocksDB instances on each TabletServer - **Resource isolation**: Prevents any single table from consuming excessive I/O bandwidth - **Cost efficiency**: Reduces memory overhead compared to per-instance rate limiters ### 2. Configuration Options **New configuration: `kv.shared-rate-limiter-bytes-per-sec`** - **Type**: MemorySize - **Default**: `0b` (disabled) - **Scope**: Cluster-level, dynamically updatable - **Example values**: `100MB`, `500MB`, `1GB` Static configuration in `conf/server.yaml`: kv.shared-rate-limiter-bytes-per-sec: 200mb### 3. Dynamic Configuration Support Rate limiter can be updated at runtime without server restart: - **Persistence**: Changes persisted to ZooKeeper - **Cluster-wide**: Applies to all TabletServers - **Zero-downtime**: Thread-safe updates via RocksDB API ### 3. Flink Stored Procedures SQL-based management interface for operators: **Set rate limiter:** -- Named argument (Flink 1.19+) CALL fluss_catalog.sys.set_shared_rocksdb_rate_limiter(rate_limit => '200MB'); -- Indexed argument CALL fluss_catalog.sys.set_shared_rocksdb_rate_limiter('500MB'); -- Disable rate limiter CALL fluss_catalog.sys.set_shared_rocksdb_rate_limiter('0MB');**Query current setting:** CALL fluss_catalog.sys.get_shared_rocksdb_rate_limiter(); ### Anything else? _No response_ ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
