Jackie-Jiang opened a new pull request, #12886: URL: https://github.com/apache/pinot/pull/12886
Currently Helix state transition is not handled in an idempotent way. The handling logic is as following: In OFFLINE table: - `OFFLINE -> ONLINE` and `REFRESH` message are handled with `addOrReplaceSegment()` In REALTIME table: - `OFFLINE -> CONSUMING` and `OFFLINE -> ONLINE` are handled with `addRealtimeSegment()` - `CONSUMING -> ONLINE` is handled with `goOnlineFromConsuming()` - `REFRESH` message is handled with `addOrReplaceSegment()` This can potentially cause the consuming segment hold lock forever in the following scenario: - Server has a `CONSUMING` segment holding the lock - Helix created a new session (e.g. ZK disconnect then reconnect) which causes segment to start from the initial state (`OFFLINE`), at the same time the segment is committed by another server - Server will get a `OFFLINE -> ONLINE` state transition, which won't change the segment because currently server cannot differentiate `OFFLINE -> CONSUMING` and `OFFLINE -> ONLINE` transition (notice that they are handled by the same method). - This will leave the segment in consuming state forever because in Helix's view the segment is already `ONLINE`, and it will never issue another `CONSUMING -> ONLINE` state transition. This PR fixes the issue by invoking the proper method based on the target state (idempotency): - `OFFLINE -> ONLINE` and `CONSUMING -> ONLINE` are both handled with `addOnlineSegment()` - `OFFLINE -> CONSUMING` is handled with `addConsumingSegment()` - `REFRESH` message is handled with `replaceSegment()` This also simplifies the handling logic a lot. ## Incompatible `InstanceDataManager` and `TableDataManager` interface are changed. They are both internal only, but in case some custom code are using these APIs it might break. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
