ryanmorris708 commented on issue #10615: URL: https://github.com/apache/druid/issues/10615#issuecomment-840846463
I am seeing this as well in version 0.20.1, deployed on Kubernetes using the [Druid Operator](https://druid.apache.org/docs/latest/operations/kubernetes.html). I tried changing the securityContext to use root for everything, since some of the files that the SysMonitor wants to read are owned by root, but this didn't help. The log is always exactly the same (always DiskUsage regardless of the file), with the same stack trace: `<date> ERROR [MonitorScheduler-0] org.apache.druid.java.util.metrics.SysMonitor - Failed to get DiskUsage[<file>]` The affected files/directories are: - /dev/termination-log - /etc/hosts - /etc/resolv.conf - /etc/hostname - /opt/druid/var - /opt/druid/conf/druid/cluster/_common - /opt/druid/conf/druid/cluster/data/broker - /opt/druid/conf/druid/cluster/data/coordinator - /opt/druid/conf/druid/cluster/data/historical - /opt/druid/conf/druid/cluster/data/middleManager - /opt/druid/conf/druid/cluster/data/router Relevant parts of my Druid resource definition (no volumes mounted yet): ``` apiVersion: "druid.apache.org/v1alpha1" kind: "Druid" metadata: name: dev spec: image: apache/druid:0.20.1 startScript: /druid.sh securityContext: fsGroup: 0 runAsUser: 0 runAsGroup: 0 services: - spec: type: LoadBalancer commonConfigMountPath: "/opt/druid/conf/druid/cluster/_common" jvm.options: |- -server -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/druid/var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager common.runtime.properties: | druid.extensions.loadList=["druid-histogram", "druid-datasketches", "postgresql-metadata-storage", "druid-kafka-indexing-service", "kafka-emitter", "druid-s3-extensions"] druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor", "org.apache.druid.java.util.metrics.SysMonitor", "org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.java.util.metrics.JvmCpuMonitor", "org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor", "org.apache.druid.java.util.metrics.JvmThreadsMonitor", "org.apache.druid.server.metrics.EventReceiverFirehoseMonitor"] ``` Example stack trace for /dev/termination-log: ``` 2021-05-13T21:09:48,627 ERROR [MonitorScheduler-0] org.apache.druid.java.util.metrics.SysMonitor - Failed to get DiskUsage[/dev/termination-log] org.hyperic.sigar.SigarFileNotFoundException: No such file or directory at org.hyperic.sigar.DiskUsage.gather(Native Method) ~[sigar-1.6.5.132.jar:?] at org.hyperic.sigar.DiskUsage.fetch(DiskUsage.java:30) ~[sigar-1.6.5.132.jar:?] at org.hyperic.sigar.Sigar.getDiskUsage(Sigar.java:680) ~[sigar-1.6.5.132.jar:?] at org.apache.druid.java.util.metrics.SysMonitor$DiskStats.emit(SysMonitor.java:310) [druid-core-0.20.1.jar:0.20.1] at org.apache.druid.java.util.metrics.SysMonitor.doMonitor(SysMonitor.java:107) [druid-core-0.20.1.jar:0.20.1] at org.apache.druid.java.util.metrics.AbstractMonitor.monitor(AbstractMonitor.java:47) [druid-core-0.20.1.jar:0.20.1] at org.apache.druid.java.util.metrics.MonitorScheduler$1.call(MonitorScheduler.java:124) [druid-core-0.20.1.jar:0.20.1] at org.apache.druid.java.util.metrics.MonitorScheduler$1.call(MonitorScheduler.java:119) [druid-core-0.20.1.jar:0.20.1] at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$4.run(ScheduledExecutors.java:158) [druid-core-0.20.1.jar:0.20.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_275] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_275] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
