[
https://issues.apache.org/jira/browse/KAFKA-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandini Singhal updated KAFKA-19968:
------------------------------------
Description:
Tiered storage fetch quotas are not properly enforced. Brokers can
significantly exceed configured fetch quota limits because the quota check only
evaluates historical usage and does not account for the size of the fetch
request that is about to be executed.
The issues are:
# Quota check happens BEFORE the fetch
# The check only considers past usage
# The fetch size is known but not considered
# Actual bytes are recorded AFTER the fetch completes
*Example Scenario:*
- Configured quota: 136 MB/s
- Current usage (past 1 second window): 100 MB/s
- New fetch request arrives requesting 50 MB
*Current behavior:*
1. getFetchThrottleTimeMs() checks: 100 MB/s < 136 MB/s → Not violated,
returns 0
2. Fetch proceeds and downloads 50 MB
3. Total throughput: 150 MB/s → Exceeds quota by 14 MB/s
4. quotaManager.record(50 MB) is called
5. Next request will be throttled, but quota has already been exceeded
*Expected behavior:*
1. Quota check should evaluate: (100 MB/s + 50 MB estimated) = 150 MB/s > 136
MB/s → Violation
2. Request should be throttled BEFORE fetching from remote storage
3. Quota should never be exceeded
was:
Tiered storage fetch quotas are not properly enforced. Brokers can
significantly exceed configured fetch quota limits because the quota check only
evaluates historical usage and does not account for the size of the fetch
request that is about to be executed.
The issues are:
# Quota check happens BEFORE the fetch
# The check only considers past usage
# The fetch size is known but not considered
# Actual bytes are recorded AFTER the fetch completes
Example Scenario:
- Configured quota: 136 MB/s
- Current usage (past 1 second window): 100 MB/s
- New fetch request arrives requesting 50 MB
Current behavior:
1. getFetchThrottleTimeMs() checks: 100 MB/s < 136 MB/s → Not violated,
returns 0
2. Fetch proceeds and downloads 50 MB
3. Total throughput: 150 MB/s → Exceeds quota by 14 MB/s
4. quotaManager.record(50 MB) is called
5. Next request will be throttled, but quota has already been exceeded
> Tiered Storage Fetch Quotas Not Enforced - Quota Check Does Not Account for
> In-Flight Fetch Size
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-19968
> URL: https://issues.apache.org/jira/browse/KAFKA-19968
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Reporter: Nandini Singhal
> Assignee: Nandini Singhal
> Priority: Major
>
> Tiered storage fetch quotas are not properly enforced. Brokers can
> significantly exceed configured fetch quota limits because the quota check
> only evaluates historical usage and does not account for the size of the
> fetch request that is about to be executed.
> The issues are:
> # Quota check happens BEFORE the fetch
> # The check only considers past usage
> # The fetch size is known but not considered
> # Actual bytes are recorded AFTER the fetch completes
> *Example Scenario:*
> - Configured quota: 136 MB/s
> - Current usage (past 1 second window): 100 MB/s
> - New fetch request arrives requesting 50 MB
> *Current behavior:*
> 1. getFetchThrottleTimeMs() checks: 100 MB/s < 136 MB/s → Not violated,
> returns 0
> 2. Fetch proceeds and downloads 50 MB
> 3. Total throughput: 150 MB/s → Exceeds quota by 14 MB/s
> 4. quotaManager.record(50 MB) is called
> 5. Next request will be throttled, but quota has already been exceeded
>
> *Expected behavior:*
> 1. Quota check should evaluate: (100 MB/s + 50 MB estimated) = 150 MB/s >
> 136 MB/s → Violation
> 2. Request should be throttled BEFORE fetching from remote storage
> 3. Quota should never be exceeded
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
