jakubmatyszewski opened a new issue, #14599: URL: https://github.com/apache/druid/issues/14599
### Affected Version 26.0.0 Problem --- Docs of `druid-overlord-extension` missing details and/or `druid.indexer.runner.peonMonitors` not working as expected. ### Description While following [instructions of MM-less `druid-overlord-extension`](https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs.html) I had a problem with peons still trying to use monitors defined in Overlord's runtime.properties. Docs say: > druid.indexer.runner.peonMonitors [JsonArray] - Overrides druid.monitoring.monitors. Use this property if you don't want to inherit monitors from the Overlord But even though my overlord runtime.properties looked like this: ``` (...) druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.TaskCountStatsMonitor"] druid.indexer.runner.peonMonitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.EventReceiverFirehoseMonitor"] ``` in logs of spawned job pods I received following error: ``` 1) No implementation for org.apache.druid.server.metrics.TaskCountStatsProvider was bound. while locating org.apache.druid.server.metrics.TaskCountStatsProvider ``` However on startup of said pods I saw that peonMonitors were registered correctly: ``` 2023-07-17T12:33:56,412 INFO [main] org.apache.druid.cli.CliOverlord - * druid.indexer.runner.peonMonitors: ["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.EventReceiverFirehoseMonitor"] ``` Workaround --- Looking into code, I've found that runtime.properties of Overlord are not overridden directly, but rather environment variable called `druid_monitoring_monitors` is emptied, peons overwrite this variable and it expects to go from here. [This is the code](https://github.com/apache/druid/blob/1f6507dd604da089fdb1a0b66f4b7eed06cc9672/extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/taskadapter/K8sTaskAdapter.java#L215C15-L215C15). Workaround for me, just for now is using environment variable in Overlord's runtime.properties like this: ``` druid.monitoring.monitors=${env:druid_monitoring_monitors:-["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.TaskCountStatsMonitor"]} ``` This way I don't have to change overlord's pod spec - it will use default array that is provided, but for peon it will be correctly emptied and applied with array passed to `druid.indexer.runner.peonMonitors` variable. Leaving it here in case someone encounters similar issue. I'm not sure if I'd manage to propose a PR to change the code, but perhaps at least I will add a notice about this in docs soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
