Vamsi-klu opened a new pull request, #19370:
URL: https://github.com/apache/pinot/pull/19370
## Problem
A retryable Kafka/DNS failure during consuming-segment init still runs on
the Helix state-transition thread. That path uses Kafka's short 5x2s retry,
then marks the replica OFFLINE after about 10s. Healthy replicas keep
consuming. The partition stays under-replicated until the next natural flush.
#17062 already retries mid-consume recreate on the consumer thread with
`CONSUMER_RECREATE_RETRY_POLICY` (about 8 minutes). First create did not use
that path.
#19083 tried a controller-side auto force-commit. Review feedback was that
one bad server must not force healthy replicas to commit, and that the
remaining hole is this init path, not another default-off controller flag.
## What I did
Move first `makeStreamConsumer("Starting")` from the Helix constructor onto
`PartitionConsumer.run()`, after the ready-wait and consumer semaphore. First
create now uses the same `CONSUMER_RECREATE_RETRY_POLICY` as mid-consume
recreate (10 attempts, 1s, factor 2).
No force-commit. No new controller config. #17754
(`controller.realtime.segment.partialOfflineReplicaRepairEnabled`, default
false) is unchanged. All-OFFLINE recreate is unchanged.
## How
- Helix constructor still builds the decoder, transform pipeline, mutable
segment, and partition metadata provider. It does not open the stream consumer.
- Exhausted init still takes the existing `postStopConsumedMsg` / ERROR /
OFFLINE path.
- `stop()` for CONSUMING to ONLINE keeps a just-created consumer so catchup
can use it. Offload during in-flight create closes the uninstalled client.
- Catchup requires both the consumer semaphore and a live consumer.
Otherwise the replica downloads.
- Close leaves the reference in place so COMMIT/CATCH_UP can still
`checkpoint` a closed instance (default is identity).
## Impact
A transient Kafka/DNS blip on one server retries on that server's consumer
thread instead of going OFFLINE after about 10s. Other replicas are not
force-committed. Brokers do not start preferring a replica that is hours behind.
Helix state-transition threads are not blocked for the long retry window.
## Not in this PR
`createPartitionMetadataProvider("Starting")` still runs on the Helix
constructor. For Kafka that opens a metadata-provider client on the short 5x2s
path. A retryable DNS failure there can still OFFLINE a replica after about
10s. Fetch fallbacks already exist for offset lookup; provider construction
itself can still throw.
## Testing
`RealtimeSegmentDataManagerTest`: 33 tests, 0 failures.
- ctor does not create a stream consumer
- first create uses `CONSUMER_RECREATE_RETRY_POLICY`
- exhaustion posts stop-consumed and goes ERROR
- offload with never-created consumer succeeds
- stop during init keeps the consumer for catchup
- offload during init closes the created client
- ONLINE with semaphore but no consumer downloads
```
./mvnw -pl pinot-core -am -Dtest=RealtimeSegmentDataManagerTest
-Dsurefire.failIfNoSpecifiedTests=false test
```
Tagged `bug` and `testing`. No `release-notes` (no new config).
## Related
- #15897
- #11314
- #17062
- #17754
- Alternative to #19083 (no force-commit, no new controller flag)
cc @noob-se7en
##### Was generative AI tooling used to co-author this PR?
- [x] Yes
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
