issam-messai opened a new issue, #17820: URL: https://github.com/apache/druid/issues/17820
Excessive Memory Usage by LookupCoordinatorManager in master nodes Due to keeping a copy of each Lookups per process ### Affected Version The Druid version where the problem was encountered is 24.0.0 ### Description I encountered a severe memory usage issue in master nodes when using lookups. The [LookupCoordinatorManager](https://github.com/apache/druid/blob/24.0.0/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java) keeps **a copy** of each static lookup in the member `knownOldState` for **each running process on query and data servers** (historical, middleManager, peon, broker, router) in the cluster. As a result, the memory consumption scales **linearly** with the number of processes, causing excessive heap usage on master nodes and `ExitOnOutOfMemoryError` crashes. ## Observed Behavior - I added lookups with a **total size of 100MB**. - My cluster has **at least running 14 processes** on data and query servers without counting peon processes. - The lookup memory consumption on the master node reached more than **3.6GB** (`100MB x 2 x 14`), leading to OOM java exception - Heap dump analysis using **Eclipse Memory Analyzer (MAT)** showed that `org.apache.druid.server.lookup.cache.LookupCoordinatorManager` is consuming 90% of the heap. (attaches some screenshots of the heap dump analysis)  ## Expected Behavior - `LookupCoordinatorManager` should **not duplicate** lookups unnecessarily for each process. - Lookups should have a **shared or optimized memory footprint** across processes. - The memory overhead for lookups should remain **proportional** to their actual size, not process count. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
