Tarun-kishore opened a new pull request, #19838:
URL: https://github.com/apache/pinot/pull/19838
## Summary
Adds support for `SHADOW` and `DISABLED` enforcement modes for storage and
query QPS quotas, allowing operators to evaluate quota limits before enforcing
them.
Currently, Pinot enforces quotas as soon as they are configured. This makes
it difficult to roll out quotas to existing tables without risking rejected
queries, failed segment uploads, or paused realtime consumption.
This change introduces a `QuotaEnforcementMode` enum with three modes:
| Mode | Behavior |
|---|---|
| `ENFORCED` | Existing behavior. Quota breaches are recorded and enforced. |
| `SHADOW` | Quota breaches are recorded, but queries, uploads, and realtime
consumption are not blocked. |
| `DISABLED` | Quota enforcement is skipped and no breach metrics are
emitted. |
`ENFORCED` remains the default when no enforcement mode is configured.
## Changes
### Configuration
Added two optional fields to `QuotaConfig` to control storage and query
quota enforcement independently:
```json
{
"quota": {
"storage": "10G",
"maxQueriesPerSecond": "100",
"storageQuotaEnforcementMode": "SHADOW",
"queryQuotaEnforcementMode": "SHADOW"
}
}
```
### Storage Quota (Controller)
- Updated storage quota checks to respect the configured enforcement mode.
- In `SHADOW` mode, segment uploads are allowed even when the storage quota
is exceeded.
- Realtime consumption is not paused for storage quota breaches, and tables
already paused due to quota can resume when switched to `SHADOW`.
- The existing `controller.enable.storage.quota.check=false` flag continues
to disable storage quota checks globally.
### Query Quota (Broker)
- Updated broker-side QPS enforcement to support the new modes.
- In `SHADOW` mode, queries exceeding the configured QPS limit are still
executed, while breaches are recorded.
- For queries using a raw table name with both OFFLINE and REALTIME tables,
the strictest enforcement mode among the existing tables is applied.
### Metrics
- Reuses `BrokerMeter.QUERY_QUOTA_EXCEEDED` for query quota breaches.
- Adds `ControllerMeter.STORAGE_QUOTA_EXCEEDED` for storage quota breaches.
- Both `SHADOW` and `ENFORCED` emit the same breach meters, allowing
operators to compare observed violations before enabling enforcement.
- The existing `TableStorageQuotaUtilization` gauge remains unchanged.
- `DISABLED` does not emit quota breach metrics.
## Backward Compatibility
- Existing table configurations without the new fields continue using
`ENFORCED`, with no change to their serialized representation.
- Older Pinot binaries ignore the new fields. If an older binary rewrites a
table config, the fields may be dropped, causing enforcement to fall back to
`ENFORCED`.
- Controllers and brokers should be upgraded before configuring the new
enforcement modes on tables.
## Testing
### Unit Tests
Added unit tests covering:
- Config serialization and default enforcement mode.
- Storage quota checker, segment upload validation, and realtime consumption
validation.
- Broker QPS enforcement and quota manager behavior.
### Integration Tests
- `QueryQuotaClusterIntegrationTest`: **16/16 passing**, including `SHADOW`,
`DISABLED`, and unset/default enforcement modes.
### Local Quick-Start Validation
| Scenario | ENFORCED | SHADOW | DISABLED |
|---|---|---|---|
| Query quota (40 rapid queries) | 38 rejected | 0 rejected | 0 rejected |
| Segment upload exceeding storage quota | HTTP 403 | HTTP 200 | HTTP 200 |
| Quota breach metrics | Emitted | Emitted | Not emitted |
Verified that `SHADOW` allows requests while still recording quota breaches,
and that existing enforcement behavior is preserved when the new fields are not
configured.
Instructions:
1. The PR has to be tagged with at least one of the following labels (*):
1. `feature`
2. `bugfix`
3. `performance`
4. `ui`
5. `backward-incompat`
6. `release-notes` (**)
2. Remove these instructions before publishing the PR.
(*) Other labels to consider:
- `testing`
- `dependencies`
- `docker`
- `kubernetes`
- `observability`
- `security`
- `code-style`
- `extension-point`
- `refactor`
- `cleanup`
(**) Use `release-notes` label for scenarios like:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
