FrankChen021 opened a new issue, #20750: URL: https://github.com/apache/druid/issues/20750
### Affected Version 34.0.0 and later: every release that includes #17935, up to the current `master`. We hit this on 38.0.0 when upgrading a cluster from 27.0.0. ### Description There are two related problems. #### 1. User-facing: a standalone Overlord never starts the segment metadata cache, and segment writes hang **Setup** - The Coordinator and Overlord run as separate services. - `druid.manager.segments.useIncrementalCache=always`, as recommended for MSQ compaction. - The Overlord's `runtime.properties` contains `druid.coordinator.asOverlord.enabled=true`. The last point is easy to hit by accident. The Docker image's `druid.sh` uses the same `conf/druid/cluster/master/coordinator-overlord` directory for both the `coordinator` and `overlord` services, and that file sets `druid.coordinator.asOverlord.enabled=true`. Helm charts built on that image inherit it unless they override the property on the Overlord as well. Before 34 the Overlord never read `enabled`, so this was harmless and nobody needed the override. It starts to matter on the first upgrade to 34 or later. **What happens** 1. Because `asOverlord.enabled` is `true`, `DruidOverlord.start()` assumes a Coordinator in the same process owns the segment metadata cache, and skips `SegmentMetadataCache.start()`. No Coordinator runs there, so the cache never syncs. 2. The first leader election fails with `Cache has not been started yet`. The next one succeeds but leaves the cache in `LEADER_FIRST_SYNC_PENDING` forever, because `stopBeingLeader()` had moved it from `STOPPED` to `FOLLOWER`. 3. Every segment write, such as `markUnused` or compaction, then waits in `HeapMemorySegmentMetadataCache.verifyCacheIsUsableAndAwaitSyncIf` with no time limit. Each one holds a Jetty thread and an open metadata-store transaction. **Observed effect** - Overlord endpoints such as `markUnused` and `/druid/indexer/v1/tasks` became slow and started returning 500s. - The Overlord's Jetty thread count kept growing. With the default `useIncrementalCache=ifSynced`, the Overlord does not hang. It still never starts the cache, so every transaction silently goes straight to the metadata store. **Workaround:** set `druid.coordinator.asOverlord.enabled=false` explicitly on the standalone Overlord. #### 2. Dev-facing: the Overlord decides its own mode from a Coordinator setting `druid.coordinator.asOverlord.enabled` is documented as a Coordinator setting: "whether this Coordinator service should act like an Overlord as well". The Overlord-side code still uses it to decide how the Overlord itself is deployed: - `DruidOverlord.isStandalone()` returns `!coordinatorOverlordServiceConfig.isEnabled()`. This decides whether the Overlord starts and stops the segment metadata cache it shares with the Coordinator. It was introduced in #17935. - `OverlordCompactionScheduler` sets `shouldPollSegments = segmentManager != null && !coordinatorOverlordServiceConfig.isEnabled()`. So a standalone Overlord that sees the property never polls segments for compaction supervisors. The process already knows its mode without reading any property. `CliOverlord.getModules(boolean standalone)` is called with `true` when running as `overlord`, and with `false` when `CliCoordinator` embeds the Overlord modules. Inferring the mode from a Coordinator property instead is what lets a shared or inherited `runtime.properties` silently break a standalone Overlord. There is related dead code. `CoordinatorOverlordServiceConfig.getOverlordService()` has had no callers since #19523, which removed its last use. Even so, the docs and example configs still describe `druid.coordinator.asOverlord.overlordService` as required when `enabled=true`, and the config class still validates that it is set. ### Proposed fix - Have `CliOverlord` bind the mode it already knows, as an `OverlordMode` enum (`STANDALONE` / `EMBEDDED_IN_COORDINATOR`). Inject it into `DruidOverlord` and `OverlordCompactionScheduler` in place of `CoordinatorOverlordServiceConfig`. - Remove `CoordinatorOverlordServiceConfig`, which is then unused. Drop `druid.coordinator.asOverlord.overlordService` from the docs and example configs. I will open a PR for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
