findingrish commented on code in PR #13967:
URL: https://github.com/apache/druid/pull/13967#discussion_r1694291192
##########
server/src/main/java/org/apache/druid/client/BrokerServerView.java:
##########
@@ -173,6 +181,37 @@ public CallbackAction segmentViewInitialized()
return CallbackAction.CONTINUE;
}
);
+
+ metadataSegmentView.registerSegmentCallback(
Review Comment:
I have added class level javadocs to explain the flow. Attaching the
explanation here as well,
```
There are two primary flows for segment management:
1. Standard Flow: Segments are added to the timeline after being loaded by
historicals.
2. Unavailable Segment Detection Flow: This flow is enabled when unavailable
segment detection is turned on.
### Segment Lifecycle
The lifecycle of a segment `s` can be described as follows:
`Creation`: Segment metadata is published in the database at time `t1`.
`Coordinator Polling`: The Coordinator polls for new segments at time `t2`
but does not consider `s` loaded yet.
`Broker Polling`: The Broker polls the Coordinator at time `t3` and finds
segment `s` but doesn't add it to the timeline as it's not loaded.
`Historical Loading`: The segment `s` is loaded by a Historical process at
time `t4`.
`Coordinator Update`: The Coordinator receives a callback from the
Historical and marks the segment as `loaded` at time `t5`.
### Segment Availability
Once marked as `loaded`, a segment is considered available for querying.
However, there are potential scenarios that can impact segment availability:
#### Scenario 1: Broker Receives Segment Metadata Before Historical Callback
* The Broker adds `s` to the timeline as `queryable` at time `t6`.
* Queries for `s` at this point might fail as it's not fully loaded.
* Once the Broker receives the callback from the Historical at time `t7`,
the segment becomes available for querying.
* If all Historicals serving `s` go down, the segment remains in the
timeline as `queryable` but becomes unavailable for querying.
#### Scenario 2: Broker Receives Historical Callback Before Segment Metadata
* The Broker adds `s` to the timeline but doesn't mark it as `queryable` at
time `t6`.
* Queries for `s` will still be processed, but the segment might not be
fully available.
* Once the Broker receives metadata from the Coordinator at time `t7`, the
segment is marked as `queryable`.
* Similar to Scenario 1, if all Historicals serving `s` go down, the segment
remains in the timeline as `queryable` but becomes unavailable for querying.
Conclusion:
The availability of a segment in the timeline doesn't guarantee its actual
availability for querying. The system relies on the timely updates from
Historicals and Coordinators to ensure correct segment status.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
