suvodeep-pyne opened a new pull request, #19737:
URL: https://github.com/apache/pinot/pull/19737
## Summary
The controller repairs pauseless segments that are stuck COMMITTING with no
download URL by calling the server's `POST /reingestSegment/{segmentName}`. On
the server, `ReingestionResource` re-consumes the segment's offset range and
waits for consumption to reach the end offset. That wait has a hard-coded
30-minute deadline (`// TODO: Make them configurable`). This PR makes the
deadline configurable, including through Helix cluster config, so an operator
can raise it without a restart.
## Motivation
A pauseless REALTIME table had several ~126–148M-row segments stuck
COMMITTING with no download URL. Every replica was in ERROR. Each re-ingestion
attempt consumed ~88M rows at ~49k rows/s and then failed at the 30-minute
deadline with `Timeout waiting for condition`. Every retry restarts from the
start offset, so a segment that can't be consumed within 30 minutes can never
be recovered by the repair path, and the limit could not be raised without a
code change.
## Changes
- New config key `pinot.server.reingestion.consumption.timeoutMs`. The
default is `1800000` (30 minutes), so default behavior is unchanged.
- The timeout is resolved when each re-ingestion request is accepted, in
this order:
1. Helix cluster config (`POST /cluster/configs`), read from the server's
live cluster-config snapshot. A change applies to the next re-ingestion job
with no restart.
2. Server config.
3. The 30-minute default.
A non-numeric or non-positive value is skipped with a WARN and resolution
falls through to the next level. A job that is already running keeps the
timeout it started with.
Note: as with other server configs,
`ServiceStartableUtils.applyClusterConfig` copies cluster configs that exist at
server startup into the server config. If the key was in cluster config when
the server started, deleting it later falls back to that startup copy, not the
default, so change the value instead of deleting the key.
- The server's `DefaultClusterConfigChangeHandler` is bound into the server
admin API as a `PinotClusterConfigProvider`, so the resource can read live
cluster configs. This adds a constructor parameter to `AdminApiApplication`;
`BaseServerStarter#createServerAdminApp()` is updated.
- The deadline computation saturates, so a very large configured value
cannot overflow.
- Removed the unused public constant
`ReingestionResource.CONSUMPTION_END_TIMEOUT_MS`.
Example, raising the limit to 2 hours cluster-wide:
```bash
curl -X POST "http://<controller>/cluster/configs" -H "Content-Type:
application/json" \
-d '{"pinot.server.reingestion.consumption.timeoutMs": "7200000"}'
```
## Backward compatibility
- The default is unchanged. There are no wire-protocol, ZK metadata or
controller changes.
- `AdminApiApplication`'s constructor takes a new
`PinotClusterConfigProvider` argument. Code that constructs it directly must
pass one; subclasses of `BaseServerStarter` that go through
`createServerAdminApp()` are unaffected.
- The removed `ReingestionResource.CONSUMPTION_END_TIMEOUT_MS` was public
but had no references.
## Testing
`ReingestionResourceTest` covers:
- the resolution order (default, server config, cluster config overriding
server config);
- invalid values falling back a level;
- boundary values (`1`, whitespace, `Long.MAX_VALUE`) and invalid values
(non-numeric, non-positive, overflowing) falling back a level;
- a live cluster-config set and remove on a single
`DefaultClusterConfigChangeHandler`;
- the deadline overflow guard;
- an endpoint test that the resource's injected dependencies are bound. It
fails if the new binding is removed.
## Out of scope (possible follow-ups)
- Re-ingestion does not log consumption progress and does not resume across
attempts.
- `MAX_PARALLEL_REINGESTIONS` is still hard-coded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
