jtuglu1 opened a new issue, #18764: URL: https://github.com/apache/druid/issues/18764
I've identified a race in segment movement which causes temporary segment unavailability (thus and partial query results). Scenario: Segment A is being moved from server A to server B. The race is as follows: 1. (Coordinator): Running dutyGroup[HistoricalManagementDuties]. 2. (Coordinator): Marking segment[S] to drop on server[A] to load on server[B]. 3. (Server B): Loading segment[S] 4. (Coordinator): Finished running dutyGroup[HistoricalManagementDuties]. 5. (Coordinator): Running dutyGroup[HistoricalManagementDuties]. 6. (Coordinator): Dropping segment [S] from server [A]. 7. (Coordinator): Using round robin segment load strategy. 8. (Server A): Completely removing segment[S]. 9. numReplicasToDrop[1], loadedNotDropping[2], loading[0], mvCompletedPendingDrop[0], segment[S], tier[_default_tier] 10. Dropping replicas for [S] with [1], [1] 11. (Coordinator): Dropping segment [S] from server [B]. (RACE) The issue is that there is a momentary lapse in time between when server B loads the segment (and it [appears](https://github.com/apache/druid/blob/-/server/src/main/java/org/apache/druid/server/coordinator/loading/SegmentReplicaCountMap.java#L54) as loaded in accounting) and when the callback for dropping on server A occurs. Because the `projectedReplicas` (calculated [here](https://github.com/apache/druid/blob/-/server/src/main/java/org/apache/druid/server/coordinator/loading/StrategicSegmentAssigner.java#L266)) is: `loadedNotDropping` + `loading` - `max(0, moveFrom - moveTo)` = 2 (both are loaded at this point) + 0 - max(0, 1 - 1) = 2, this causes the drop to be subsequently scheduled on B. ### Affected Version All recent Druid versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
