xiangfu0 opened a new pull request, #20017: URL: https://github.com/apache/pinot/pull/20017
The broker routes a new segment as optional until it sees the segment ONLINE in the external view. The single-stage engine sends optional segments to the servers, which query them if loaded and skip them otherwise. Multi-stage leaf stages dropped them, so right after a segment upload or a realtime commit a multi-stage query could return fewer rows than the single-stage one, with no error (the WindowFunnelTest flake worked around in #20015). Leaf workers now carry their optional segments apart from their segments, and the server leaf sets them as `InstanceRequest.optionalSegments` (`TableSegmentsInfo.optionalSegments` for logical tables). They are not merged into the segment list as in #19144: a segment that is not loaded yet would then fail the query as missing. - Covered: non-partitioned leaf stages of physical and logical tables, on both the default planner and the physical optimizer. - Unchanged: partitioned leaf stages, and replicated leaf stages (their segment list does not come from instance selection). - Wire: new `WorkerMetadata.optionalSegmentsMap` proto field, or an `optionalSegmentsMap` custom property under the legacy JSON encoding. An older server ignores either and skips optional segments as before; an older broker sends none. - `WorkerManager#filterLeafStageSegments` overrides see the optional segments in a separate map (`DispatchablePlanMetadata#getWorkerIdToOptionalSegmentsMap`). Testing: new `OptionalSegmentsQueryRunnerTest` (a loaded optional segment is queried and a missing one is skipped without error, on both planners; returns half the rows without this change), plus planner, serde (both encodings) and `ServerPlanRequestUtils` unit tests. Related: #18223, #19537. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
