clintropolis opened a new pull request, #19461: URL: https://github.com/apache/druid/pull/19461
### Description Follow-up to #19409, this PR enhances the coordintor side stuff so that partial replication is wired up and managed, taking the partial load size and rule fingerprints into consideration when doing assignment and balancing. A lot of the new code is similar to existing functionality of `StrategicSegmentAssigner`, but separate to minimize the chance of impacting existing code paths. changes: * adds selective inventory wrapping with `DataSegmentAndLoadProfile` which extends `DataSegment`; full-load slots in `DruidDataSource` / `ImmutableDruidDataSource` stay bare, only partial-loaded slots allocate a wrapper * `PartialLoadProfile` records routed through a weak interner so identical profiles across replicas share by reference (the `wrappedLoadSpec` map is the bulk of per-replica heap). * `SegmentChangeRequestLoad.forAnnouncement(segment)` detects a `partialProjection` wrapper on the segment's load spec and stamps the wrapper's fingerprint plus full segment size as `loadedBytes` — the coordinator reads this as a full-fallback profile, so partial-load rules work end-to-end (modulo no real partial loading yet) without thrashing * `BatchDataSegmentAnnouncer` calls new method `forAnnouncement` on all three announcement paths to create the proper announcement depending on partial load profile * fingerprint-aware partial reconciler in `StrategicSegmentAssigner`; classifies replicas (matching/stale, loaded/in-flight) via new `PartialSegmentStatusInTier`, applies the load-then-drop swap (stale stays serving until matching has actually loaded), cancels stale in-flight loads and re-targets the freed slots, allows additive reload on stale-loaded servers to mitigate the no-spare-server stuck state * `ImmutableDruidServer.getPartialLoadProfile`, `DruidServer.addDataSegment(segment, profile)` with `loadedBytes`-aware `currSize` accounting, `HttpServerInventoryView` threads announcement-side profile through * in-flight profile tracking on `ServerHolder.startOperation(action, segment, profile)` and `SegmentHolder.getProfile()` so the reconciler can read in-flight fingerprints. * new coordinator stats - `PARTIAL_ASSIGNED`, `PARTIAL_STALE_DROPPED`, `PARTIAL_STALE_CANCELLED` * removed `SegmentLoadingCapabilities.supportsPartialLoad` (now unused — per-replica `isFullFallback` is the better signal); coordinator always issues partial-load requests, historical handles via full-fallback until real partial loading lands * tests: `StrategicSegmentAssignerPartialTest` (reconciler scenarios), `DruidServerPartialLoadTest` (size accounting), `PartialLoadProfileTest` (interning), `SegmentChangeRequestLoadTest` (forAnnouncement), updates across affected fixtures -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
