rohityadav1993 commented on PR #17754: URL: https://github.com/apache/pinot/pull/17754#issuecomment-3960731975
> 1. Can you add details on what is the current behaviour ? I believe it retries upto 5 times to create the consumer and once failed, it sends a `segmentStoppedConsuming` back to controller. Why cant increasing this retries or make it perpetual also not solve this issue ? This is right, 5 retries with 2 seconds delay, so in 10 seconds we exhaust all attempts to recover. This happens in [RealtimeSegmentDataManager.java#L1793-L1805](https://github.com/apache/pinot/blob/9de73ec4/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java#L1793-L1805) We can only have limited retries here to avoid indefiinitely waiting for any permanent underlying errors. Controller validation job is post these retries exhaust. > 2. If the underlying issue persists, we have a continuous loop of : > Validation → Repair (OFFLINE→CONSUMING) → Helix triggers transition → > Server tries to consume → Fails → segmentStoppedConsuming → OFFLINE → > Next Validation → Repair again. Im wondering what sort of race conditions can be caused by this ? And whether the server trying to start the consumption perpetually on its own and publishing a metric of retries is a safer fix ? This should be a fairly safe scenario as we are not committing/creating new segment with new metadata but basically resurrecting an offline segment to consuming only if segement is in IN_PROGRESS state in ZK metadata. > 3. To prevent (2) happening frequently, do we wanna add a lastRepairTime - and send a new event only after x seconds/minutes ? The periodicity of RealtimeSegmentValidationManager is configrable, default: 3600 so this wouldn't be too aggressive if configured correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
