kfaraz commented on PR #19252: URL: https://github.com/apache/druid/pull/19252#issuecomment-5236534285
Thanks for enabling the cache by default, @cecemei ! I think we need to call out this item in Druid 37 upgrade notes since operators might need to increase the memory of their Overlord. @317brian , could you advise on the best course of action for fixing up the upgrade notes? Should we create a PR against `druid-website-src` or just wait for Druid 38 release? --- ## Claude generated Overlord memory estimate <details> <summary>Claude provided the following analysis as a starting point for computing required increase in Overlord memory. It seems reasonable to me for the most part.</summary> __*AI generated analysis below:*__ ### Per-Segment Memory Cost Each used segment is stored as a `DataSegmentPlus` value in a `HashMap<SegmentId, DataSegmentPlus>`. The dominant cost is the `DataSegment` payload. | Component | Size | |---|---| | `DataSegmentPlus` object header + fields | ~80 bytes | | `DataSegment` (nested) — interval, version, partition num, dimensions list, metrics list, binary version, load spec map | **~800–2,000 bytes** | | `SegmentId` key — string-backed, encodes datasource/interval/version/partition | ~150–250 bytes | | 2× `DateTime` objects (`createdDate`, `usedStatusLastUpdatedDate`) | ~96 bytes | | `Boolean used` (boxed) | ~16 bytes | | `schemaFingerprint` String (~32 hex chars) | ~96 bytes | | `Long numRows` (boxed) | ~24 bytes | | `upgradedFromSegmentId` (usually `null`) | ~0 bytes | | `HashMap.Entry` node overhead | ~48 bytes | **Realistic per-segment total: ~1,300–2,500 bytes** The `DataSegment` payload is the dominant factor — it includes all JSON-deserialized fields: dimensions and metrics lists, load spec, shard spec, and all string fields. A typical segment with 10 dimensions occupies ~1–2 KB on the heap. --- ### Structural Overhead This is small relative to segment data and can generally be ignored for planning purposes. | Structure | Per-unit cost | Notes | |---|---|---| | `ConcurrentHashMap` entry | ~80 bytes | One per datasource | | `TreeMap` entry | ~48 bytes | One per unique interval | | `SegmentsInInterval` object | ~64 bytes | One per unique interval | | `HashMap` backing array | +33% capacity overhead | Due to 0.75 load factor | --- ### Quick Estimation Formula ``` Memory ≈ N_segments × 2 KB (conservative baseline) ``` | Segment count | Estimated heap | |---|---| | 100K | ~200 MB | | 500K | ~1 GB | | 1M | ~2 GB | | 2M | ~4 GB | Adjust the multiplier based on your workload: - **~1.3 KB/segment** — simple segments, few dimensions/metrics, short string values - **~2.5 KB/segment** — many dimensions/metrics, complex load specs (e.g., long S3 paths) --- ### Caveats - **`schemaFingerprint` and `numRows`** are only populated when `SegmentSchemaCache` is enabled (`druid.segmentMetadata.cache.useSchemaCache=true`). Without schema caching these fields are `null` and the per-segment cost is slightly lower. - **Pending segments** add overhead proportional to active ingestion tasks, not the total segment count — negligible at steady state. - **Recently-unused segments** (`unusedSegmentIdToUpdatedTime`) are transient: they exist only between a mark-unused operation and the next sync cycle (10-second buffer window), so they don't contribute meaningfully at steady state. - **Unused segments are not cached** — the cache explicitly excludes them (`findHighestUnusedSegmentId`, `findUnusedSegments`, etc. all throw `DruidException.defensive(...)`). --- ### Observability The cache emits the following metrics, which can be used to validate estimates against actual memory usage: | Metric | Description | |---|---| | `segment/cache/used` | Number of used segments cached, per datasource | | `segment/cache/pending` | Number of pending segments cached, per datasource | | `segment/cache/intervals` | Number of intervals cached, per datasource | | `segment/cache/syncDuration/millis` | Time taken for each sync cycle | </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
