Aight .. that fixed it for me too, was about to test it when i decided to check in here first, so you were just faster :p
If prometheus should or not run as root ... I am not sure either ... I think its a common practice to run stuff as root inside the containers ... Cadvisor seems to be running as root ... but i am not entirely sure on this one, so take my words with a grain of salt. It should be however ... at least documented with a warning ... its quite late here already but i can do it early in the morning tomorrow ... if any of you has the chance in the meantime to try and verify this ... there are some more tests that come to my mind ... Could it be that docker is not letting "nobody" read the sock? Maybe Trying running Prometheus as another user instead of nobody or root? If nobody tries this i can try it tomorrow and maybe open PR to documentation with info about this. Kind regards. On Sunday, November 15, 2020 at 11:53:50 PM UTC+1 [email protected] wrote: > This worked for me , although i'm not sure we should be running prometheus > as root > > version: '3.7' > > services: > prometheus: > image: prom/prometheus:v2.21.0 > > command: > - '--config.file=/etc/prometheus/prometheus.yml' > - '--storage.tsdb.path=/prometheus' > - '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-48h}' > user: root > volumes: > - /var/run/docker.sock:/var/run/docker.sock:ro > - ./prometheus.yml:/etc/prometheus/prometheus.yml > ports: > - target: 9090 > published: 9090 > mode: ingress > deploy: > labels: > - prometheus-job=prometheus > > mode: replicated > replicas: 1 > resources: > limits: > memory: 2048M > reservations: > memory: 512M > > On Sun, Nov 15, 2020 at 11:52 PM Kimo <[email protected]> wrote: > >> Hello, >> I've been facing the exact same issue today and its driving me equally >> crazy. I tried running prometheus as root but still: >> >> level=error ts=2020-11-15T21:45:35.983Z caller=refresh.go:98 >> component="discovery manager scrape" discovery=dockerswarm msg="Unable to >> refresh target groups" err="error while listing swarm *services*: Cannot >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker >> daemon running?" >> level=error ts=2020-11-15T21:45:35.984Z caller=refresh.go:98 >> component="discovery manager scrape" discovery=dockerswarm msg="Unable to >> refresh target groups" err="error while listing swarm *nodes*: Cannot >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker >> daemon running?" >> >> I think I've exhausted all the options I could try by myself and would >> gladly appreciate any help at this point. >> Le dimanche 15 novembre 2020 à 21:54:26 UTC+1, Julien Pivotto a écrit : >> >>> Can you run prometheus as nobody:docker? >>> On 15 Nov 12:23, Carlos Colaço wrote: >>> > sorry .. also tried changing the permissions which changed nothing... >>> > >>> > ``` >>> > # chmod +r /var/run/docker.sock >>> > # ls -la /var/run/docker.sock >>> > srw-rw-r--. 1 root docker 0 Nov 15 20:12 /var/run/docker.sock >>> > # docker service update --force monitor_private >>> > ``` >>> > >>> > On Sunday, November 15, 2020 at 9:20:07 PM UTC+1 Carlos Colaço wrote: >>> > >>> > > Hi all .. Having the same issue... >>> > > >>> > > https://github.com/prometheus/prometheus/issues/8185 >>> > > >>> > > >>> > > Also don't think changing permissions on docker sock is a good >>> option .. >>> > > that way you are giving permissions to anyone to access it and that >>> is >>> > > something not desirable ... >>> > > >>> > > What i also tried to do instead ... since prometheus runs as Nobody >>> ( uid: >>> > > 65534 ) ... i added it to the Docker group which changed nothing =/ >>> > > >>> > > Any hints or solutions for this? driving me crazy trying different >>> > > approaches and solutions.. nothing seems to work ... >>> > > >>> > > On Tuesday, August 11, 2020 at 7:03:12 AM UTC+2 [email protected] >>> wrote: >>> > > >>> > >> Thanks Julien and Tom, >>> > >> >>> > >> I got the problem which i was facing, actually when we change the >>> > >> permissions to read-write for docker.sock, permissions are only >>> changed >>> > >> till the docker daemon or docker service is restarted. Once the >>> > >> docker/daemon is restarted then the permissions for docker sock >>> changes >>> > >> back to the original one. >>> > >> >>> > >> Is there any way using which we can make permanent changes for the >>> > >> permission of docker.sock or do we need to file a issue for the >>> same, as >>> > >> docker/daemon might be restarted for various reasons >>> > >> >>> > >> >>> > >> ? >>> > >> >>> > >> On Monday, 10 August 2020 12:40:17 UTC+5:30, Umang Goel wrote: >>> > >>> >>> > >>> Hello Julien, >>> > >>> >>> > >>> group_add is not allowed in docker swarm. Do you have any other >>> > >>> workaround for this? >>> > >>> >>> > >>> -- >>> > >>> Umang >>> > >>> >>> > >>> On Monday, 10 August 2020 12:20:51 UTC+5:30, Julien Pivotto wrote: >>> > >>>> >>> > >>>> >>> > >>>> Can you use: >>> > >>>> >>> > >>>> --group-add docker? >>> > >>>> >>> > >>>> or in compose v2 file: >>> > >>>> >>> > >>>> version: "2.4" >>> > >>>> services: >>> > >>>> prometheus: >>> > >>>> group_add: >>> > >>>> - docker >>> > >>>> >>> > >>>> >>> > >>>> On 09 Aug 22:48, Umang Goel wrote: >>> > >>>> > ls -l /var/run/docker.sock >>> > >>>> > >>> > >>>> > - srwxrw-rw- 1 root docker 0 Aug 7 11:31 /var/run/docker.sock >>> > >>>> after >>> > >>>> > making changes as per Tom, >>> > >>>> > >>> > >>>> > On Sunday, 9 August 2020 02:16:28 UTC+5:30, Julien Pivotto >>> wrote: >>> > >>>> > > >>> > >>>> > > On 07 Aug 04:36, Umang Goel wrote: >>> > >>>> > > > Hello Tom, >>> > >>>> > > > >>> > >>>> > > > Even this is not working, I am still facing the same issue. >>> Can >>> > >>>> you help >>> > >>>> > > me >>> > >>>> > > > how did you implement it. >>> > >>>> > > >>> > >>>> > > >>> > >>>> > > What are you current permissions on the /var/run/docker.sock >>> ? >>> > >>>> > > >>> > >>>> > > ls -l /var/run/docker.sock >>> > >>>> > > >>> > >>>> > > > >>> > >>>> > > > On Friday, 7 August 2020 16:47:23 UTC+5:30, Tom Kun wrote: >>> > >>>> > > > > >>> > >>>> > > > > Hello Umang, >>> > >>>> > > > > >>> > >>>> > > > > What are you current permissions on the >>> /var/run/docker.sock ? >>> > >>>> > > > > >>> > >>>> > > > > I faced the same issue, and to start and no rebuild the >>> > >>>> Prometheus >>> > >>>> > > image >>> > >>>> > > > > with the appropriate user. >>> > >>>> > > > > I put the rights to read and write the docker.socket. >>> > >>>> > > > > >>> > >>>> > > > > sudo chmod 766 /var/run/docker.sock >>> > >>>> > > > > >>> > >>>> > > > > I hope this gonna help you. >>> > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > On Friday, 7 August 2020 11:59:32 UTC+2, Umang Goel >>> wrote: >>> > >>>> > > > >> >>> > >>>> > > > >> Hello Community, >>> > >>>> > > > >> >>> > >>>> > > > >> I tired using Docker Swarm Service Discovery in >>> prometheus, >>> > >>>> but >>> > >>>> > > facing >>> > >>>> > > > >> problems using it. I followed the docker swarm support >>> > >>>> documentation >>> > >>>> > > > >> <https://prometheus.io/docs/guides/dockerswarm/>. >>> Created a >>> > >>>> > > daemon.json >>> > >>>> > > > >> file and mounted /var/run/docker.sock in prometheus >>> container. >>> > >>>> > > Container is >>> > >>>> > > > >> giving permission denied error as prometheus is running >>> as >>> > >>>> nobody and >>> > >>>> > > > >> doesn't have access to mounted /var/run/docker.sock. >>> Below is >>> > >>>> my >>> > >>>> > > > >> prometheus.yml. >>> > >>>> > > > >> Prometheus Version : v2.20.1 >>> > >>>> > > > >> >>> > >>>> > > > >> prometheus: >>> > >>>> > > > >> image: prom/prometheus >>> > >>>> > > > >> networks: >>> > >>>> > > > >> - monitor >>> > >>>> > > > >> ports: >>> > >>>> > > > >> - "9090:9090" >>> > >>>> > > > >> command: >>> > >>>> > > > >> - '--config.file=/etc/prometheus/prometheus.yml' >>> > >>>> > > > >> - '--storage.tsdb.path=/prometheus' >>> > >>>> > > > >> - >>> > >>>> '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-24h}' >>> > >>>> > > > >> volumes: >>> > >>>> > > > >> - prometheus:/prometheus >>> > >>>> > > > >> - /home/efs/devops/dsm:/etc/prometheus:ro >>> > >>>> > > > >> - /var/run/docker.sock:/var/run/docker.sock:ro >>> > >>>> > > > >> deploy: >>> > >>>> > > > >> mode: replicated >>> > >>>> > > > >> replicas: 1 >>> > >>>> > > > >> resources: >>> > >>>> > > > >> limits: >>> > >>>> > > > >> memory: 1024M >>> > >>>> > > > >> reservations: >>> > >>>> > > > >> memory: 128M >>> > >>>> > > > >> >>> > >>>> > > > >> Prometheus.yml >>> > >>>> > > > >> >>> > >>>> > > > >> scrape_configs: >>> > >>>> > > > >> - job_name: 'docker' >>> > >>>> > > > >> dockerswarm_sd_configs: >>> > >>>> > > > >> - host: unix:///var/run/docker.sock >>> > >>>> > > > >> role: nodes >>> > >>>> > > > >> >>> > >>>> > > > >> Error: >>> > >>>> > > > >> [email protected] >>> > >>>> <javascript:> | >>> > >>>> > > level=error >>> > >>>> > > > >> ts=2020-08-06T07:21:19.106Z caller=refresh.go:98 >>> > >>>> component="discovery >>> > >>>> > > > >> manager scrape" discovery=dockerswarm msg="Unable to >>> refresh >>> > >>>> target >>> > >>>> > > groups" >>> > >>>> > > > >> err="error while listing swarm nodes: Got permission >>> denied >>> > >>>> while >>> > >>>> > > trying to >>> > >>>> > > > >> connect to the Docker daemon socket at >>> > >>>> unix:///var/run/docker.sock: >>> > >>>> > > Get >>> > >>>> > > > >> \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\": dial >>> unix >>> > >>>> > > > >> /var/run/docker.sock: connect: permission denied >>> > >>>> > > > >> >>> > >>>> > > > > >>> > >>>> > > > >>> > >>>> > > > -- >>> > >>>> > > > You received this message because you are subscribed to the >>> > >>>> Google >>> > >>>> > > Groups "Prometheus Users" group. >>> > >>>> > > > To unsubscribe from this group and stop receiving emails >>> from it, >>> > >>>> send >>> > >>>> > > an email to [email protected] <javascript:>. >>> > >>>> > > > To view this discussion on the web visit >>> > >>>> > > >>> > >>>> >>> https://groups.google.com/d/msgid/prometheus-users/e5e55a73-7cc1-4c0c-99e3-0a09270df62bo%40googlegroups.com. >>> >>> >>> > >>>> >>> > >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > >>>> > > -- >>> > >>>> > > Julien Pivotto >>> > >>>> > > @roidelapluie >>> > >>>> > > >>> > >>>> > >>> > >>>> > -- >>> > >>>> > You received this message because you are subscribed to the >>> Google >>> > >>>> Groups "Prometheus Users" group. >>> > >>>> > To unsubscribe from this group and stop receiving emails from >>> it, >>> > >>>> send an email to [email protected]. >>> > >>>> > To view this discussion on the web visit >>> > >>>> >>> https://groups.google.com/d/msgid/prometheus-users/e5614621-f57a-466e-befd-269bf77d69c8o%40googlegroups.com. >>> >>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> -- >>> > >>>> Julien Pivotto >>> > >>>> @roidelapluie >>> > >>>> >>> > >>> >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/e058c64f-3db4-45c2-9550-c8db557d2a2cn%40googlegroups.com. >>> >>> >>> >>> >>> -- >>> Julien Pivotto >>> @roidelapluie >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Alexandru Duzsardi, > *DevOps Engineer* > *Skype:* alexinno83 > *GPG/PGP Key*: https://keybase.io/aduzsardi/pgp_keys.asc > *GitLab:* https://gitlab.com/aduzsardi > *GitHub:* https://github.com/aduzsardi > *LinkedIn:* https://www.linkedin.com/in/aduzsardi > *E-mail:* [email protected] > > InFinIT Partners, > *Address:* Str. Macinului Nr. 17, Cluj-Napoca, Romania > *Web:* www.infinitpartners.com > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f1962c97-b545-47e8-9877-3482bdfaadean%40googlegroups.com.
