bsyk commented on issue #19833: URL: https://github.com/apache/druid/issues/19833#issuecomment-5146411719
Confirmed fixed — thanks for the pointer. Verified on a build of current `master` (`39.0.0-SNAPSHOT`) with three ad-hoc `compact` tasks against one datasource, `-XX:ActiveProcessorCount=4` throughout, varying only the task context: | `druid.indexer.fork.property.druid.segmentCache.virtualStorageLoadThreads` | `StorageLoadingThreadPool` log line | |---|---| | *(unset)* | `max concurrent on demand loads: [32]` | | `64` | `max concurrent on demand loads: [64]` | | `3` | `max concurrent on demand loads: [3]` | `[3]` is below the `Math.max(32, 4 * availableProcessors)` floor, so it isn't a default landing there by coincidence — the property is genuinely reaching the pool now. On 37 all three would have been `[8]`. One refinement to your analysis: the behaviour I reported is fixed by **#19658 (38.0.0)**, which moved sizing to the shared `@EphemeralStorageLoading` pool provided by `StorageNodeModule` from the bound `druid.segmentCache` config. The `manufacturate()` change you describe — deriving the per-task config via `segmentLoaderConfig.toEphemeralVirtualStorage().toBuilder()` — is **#19739 (39.0.0)**, not #19658. That split leaves a smaller gap on 38: there, `manufacturate()` still calls `new SegmentLoaderConfig()` and sets only `locations`, `virtualStorage`, and `virtualStorageIsEphemeral`. So `virtualStorageLoadThreads` works on 38 (pool sizing left the factory), but `virtualStorageCoalesceGapBytes`, `virtualStorageMaxFetchRunBytes`, and `virtualStorageMetadataReservationEstimate` are still silently ignored for tasks on 38 — `SegmentLocalCacheManager` reads all three off that per-task config. Those only close on 39 via #19739. Closing this as fixed in 38.0.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
