Nandini Singhal created KAFKA-19968:
---------------------------------------
Summary: Tiered Storage Fetch Quotas Not Enforced - Quota Check
Does Not Account for In-Flight Fetch Size
Key: KAFKA-19968
URL: https://issues.apache.org/jira/browse/KAFKA-19968
Project: Kafka
Issue Type: Bug
Components: Tiered-Storage
Reporter: Nandini Singhal
Tiered storage fetch quotas are not properly enforced. Brokers can
significantly exceed configured fetch quota limits because the quota check only
evaluates historical usage and does not account for the size of the fetch
request that is about to be executed.
The issues are:
# Quota check happens BEFORE the fetch
# The check only considers past usage
# The fetch size is known but not considered
# Actual bytes are recorded AFTER the fetch completes
Example Scenario:
- Configured quota: 136 MB/s
- Current usage (past 1 second window): 100 MB/s
- New fetch request arrives requesting 50 MB
Current behavior:
1. getFetchThrottleTimeMs() checks: 100 MB/s < 136 MB/s → Not violated,
returns 0
2. Fetch proceeds and downloads 50 MB
3. Total throughput: 150 MB/s → Exceeds quota by 14 MB/s
4. quotaManager.record(50 MB) is called
5. Next request will be throttled, but quota has already been exceeded
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
