On 15 Nov 16:30, Carlos Colaço wrote: > Aight .. that fixed it for me too, was about to test it when i decided to > check in here first, so you were just faster :p > > > If prometheus should or not run as root ... I am not sure either ... I > think its a common practice to run stuff as root inside the containers ... > Cadvisor seems to be running as root ... but i am not entirely sure on > this one, so take my words with a grain of salt. > > It should be however ... at least documented with a warning ... its quite > late here already but i can do it early in the morning tomorrow ... if any > of you has the chance in the meantime to try and verify this ... there are > some more tests that come to my mind ... > > Could it be that docker is not letting "nobody" read the sock? Maybe Trying > running Prometheus as another user instead of nobody or root? > > If nobody tries this i can try it tomorrow and maybe open PR to > documentation with info about this.
I guess it all depends on your distribution and how you run docker. Can you explain more your setup? > > Kind regards. > > On Sunday, November 15, 2020 at 11:53:50 PM UTC+1 > [email protected] wrote: > > > This worked for me , although i'm not sure we should be running prometheus > > as root > > > > version: '3.7' > > > > services: > > prometheus: > > image: prom/prometheus:v2.21.0 > > > > command: > > - '--config.file=/etc/prometheus/prometheus.yml' > > - '--storage.tsdb.path=/prometheus' > > - '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-48h}' > > user: root > > volumes: > > - /var/run/docker.sock:/var/run/docker.sock:ro > > - ./prometheus.yml:/etc/prometheus/prometheus.yml > > ports: > > - target: 9090 > > published: 9090 > > mode: ingress > > deploy: > > labels: > > - prometheus-job=prometheus > > > > mode: replicated > > replicas: 1 > > resources: > > limits: > > memory: 2048M > > reservations: > > memory: 512M > > > > On Sun, Nov 15, 2020 at 11:52 PM Kimo <[email protected]> wrote: > > > >> Hello, > >> I've been facing the exact same issue today and its driving me equally > >> crazy. I tried running prometheus as root but still: > >> > >> level=error ts=2020-11-15T21:45:35.983Z caller=refresh.go:98 > >> component="discovery manager scrape" discovery=dockerswarm msg="Unable to > >> refresh target groups" err="error while listing swarm *services*: Cannot > >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker > >> daemon running?" > >> level=error ts=2020-11-15T21:45:35.984Z caller=refresh.go:98 > >> component="discovery manager scrape" discovery=dockerswarm msg="Unable to > >> refresh target groups" err="error while listing swarm *nodes*: Cannot > >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker > >> daemon running?" > >> > >> I think I've exhausted all the options I could try by myself and would > >> gladly appreciate any help at this point. > >> Le dimanche 15 novembre 2020 à 21:54:26 UTC+1, Julien Pivotto a écrit : > >> > >>> Can you run prometheus as nobody:docker? > >>> On 15 Nov 12:23, Carlos Colaço wrote: > >>> > sorry .. also tried changing the permissions which changed nothing... > >>> > > >>> > ``` > >>> > # chmod +r /var/run/docker.sock > >>> > # ls -la /var/run/docker.sock > >>> > srw-rw-r--. 1 root docker 0 Nov 15 20:12 /var/run/docker.sock > >>> > # docker service update --force monitor_private > >>> > ``` > >>> > > >>> > On Sunday, November 15, 2020 at 9:20:07 PM UTC+1 Carlos Colaço wrote: > >>> > > >>> > > Hi all .. Having the same issue... > >>> > > > >>> > > https://github.com/prometheus/prometheus/issues/8185 > >>> > > > >>> > > > >>> > > Also don't think changing permissions on docker sock is a good > >>> option .. > >>> > > that way you are giving permissions to anyone to access it and that > >>> is > >>> > > something not desirable ... > >>> > > > >>> > > What i also tried to do instead ... since prometheus runs as Nobody > >>> ( uid: > >>> > > 65534 ) ... i added it to the Docker group which changed nothing =/ > >>> > > > >>> > > Any hints or solutions for this? driving me crazy trying different > >>> > > approaches and solutions.. nothing seems to work ... > >>> > > > >>> > > On Tuesday, August 11, 2020 at 7:03:12 AM UTC+2 [email protected] > >>> wrote: > >>> > > > >>> > >> Thanks Julien and Tom, > >>> > >> > >>> > >> I got the problem which i was facing, actually when we change the > >>> > >> permissions to read-write for docker.sock, permissions are only > >>> changed > >>> > >> till the docker daemon or docker service is restarted. Once the > >>> > >> docker/daemon is restarted then the permissions for docker sock > >>> changes > >>> > >> back to the original one. > >>> > >> > >>> > >> Is there any way using which we can make permanent changes for the > >>> > >> permission of docker.sock or do we need to file a issue for the > >>> same, as > >>> > >> docker/daemon might be restarted for various reasons > >>> > >> > >>> > >> > >>> > >> ? > >>> > >> > >>> > >> On Monday, 10 August 2020 12:40:17 UTC+5:30, Umang Goel wrote: > >>> > >>> > >>> > >>> Hello Julien, > >>> > >>> > >>> > >>> group_add is not allowed in docker swarm. Do you have any other > >>> > >>> workaround for this? > >>> > >>> > >>> > >>> -- > >>> > >>> Umang > >>> > >>> > >>> > >>> On Monday, 10 August 2020 12:20:51 UTC+5:30, Julien Pivotto wrote: > >>> > >>>> > >>> > >>>> > >>> > >>>> Can you use: > >>> > >>>> > >>> > >>>> --group-add docker? > >>> > >>>> > >>> > >>>> or in compose v2 file: > >>> > >>>> > >>> > >>>> version: "2.4" > >>> > >>>> services: > >>> > >>>> prometheus: > >>> > >>>> group_add: > >>> > >>>> - docker > >>> > >>>> > >>> > >>>> > >>> > >>>> On 09 Aug 22:48, Umang Goel wrote: > >>> > >>>> > ls -l /var/run/docker.sock > >>> > >>>> > > >>> > >>>> > - srwxrw-rw- 1 root docker 0 Aug 7 11:31 /var/run/docker.sock > >>> > >>>> after > >>> > >>>> > making changes as per Tom, > >>> > >>>> > > >>> > >>>> > On Sunday, 9 August 2020 02:16:28 UTC+5:30, Julien Pivotto > >>> wrote: > >>> > >>>> > > > >>> > >>>> > > On 07 Aug 04:36, Umang Goel wrote: > >>> > >>>> > > > Hello Tom, > >>> > >>>> > > > > >>> > >>>> > > > Even this is not working, I am still facing the same issue. > >>> Can > >>> > >>>> you help > >>> > >>>> > > me > >>> > >>>> > > > how did you implement it. > >>> > >>>> > > > >>> > >>>> > > > >>> > >>>> > > What are you current permissions on the /var/run/docker.sock > >>> ? > >>> > >>>> > > > >>> > >>>> > > ls -l /var/run/docker.sock > >>> > >>>> > > > >>> > >>>> > > > > >>> > >>>> > > > On Friday, 7 August 2020 16:47:23 UTC+5:30, Tom Kun wrote: > >>> > >>>> > > > > > >>> > >>>> > > > > Hello Umang, > >>> > >>>> > > > > > >>> > >>>> > > > > What are you current permissions on the > >>> /var/run/docker.sock ? > >>> > >>>> > > > > > >>> > >>>> > > > > I faced the same issue, and to start and no rebuild the > >>> > >>>> Prometheus > >>> > >>>> > > image > >>> > >>>> > > > > with the appropriate user. > >>> > >>>> > > > > I put the rights to read and write the docker.socket. > >>> > >>>> > > > > > >>> > >>>> > > > > sudo chmod 766 /var/run/docker.sock > >>> > >>>> > > > > > >>> > >>>> > > > > I hope this gonna help you. > >>> > >>>> > > > > > >>> > >>>> > > > > > >>> > >>>> > > > > On Friday, 7 August 2020 11:59:32 UTC+2, Umang Goel > >>> wrote: > >>> > >>>> > > > >> > >>> > >>>> > > > >> Hello Community, > >>> > >>>> > > > >> > >>> > >>>> > > > >> I tired using Docker Swarm Service Discovery in > >>> prometheus, > >>> > >>>> but > >>> > >>>> > > facing > >>> > >>>> > > > >> problems using it. I followed the docker swarm support > >>> > >>>> documentation > >>> > >>>> > > > >> <https://prometheus.io/docs/guides/dockerswarm/>. > >>> Created a > >>> > >>>> > > daemon.json > >>> > >>>> > > > >> file and mounted /var/run/docker.sock in prometheus > >>> container. > >>> > >>>> > > Container is > >>> > >>>> > > > >> giving permission denied error as prometheus is running > >>> as > >>> > >>>> nobody and > >>> > >>>> > > > >> doesn't have access to mounted /var/run/docker.sock. > >>> Below is > >>> > >>>> my > >>> > >>>> > > > >> prometheus.yml. > >>> > >>>> > > > >> Prometheus Version : v2.20.1 > >>> > >>>> > > > >> > >>> > >>>> > > > >> prometheus: > >>> > >>>> > > > >> image: prom/prometheus > >>> > >>>> > > > >> networks: > >>> > >>>> > > > >> - monitor > >>> > >>>> > > > >> ports: > >>> > >>>> > > > >> - "9090:9090" > >>> > >>>> > > > >> command: > >>> > >>>> > > > >> - '--config.file=/etc/prometheus/prometheus.yml' > >>> > >>>> > > > >> - '--storage.tsdb.path=/prometheus' > >>> > >>>> > > > >> - > >>> > >>>> '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-24h}' > >>> > >>>> > > > >> volumes: > >>> > >>>> > > > >> - prometheus:/prometheus > >>> > >>>> > > > >> - /home/efs/devops/dsm:/etc/prometheus:ro > >>> > >>>> > > > >> - /var/run/docker.sock:/var/run/docker.sock:ro > >>> > >>>> > > > >> deploy: > >>> > >>>> > > > >> mode: replicated > >>> > >>>> > > > >> replicas: 1 > >>> > >>>> > > > >> resources: > >>> > >>>> > > > >> limits: > >>> > >>>> > > > >> memory: 1024M > >>> > >>>> > > > >> reservations: > >>> > >>>> > > > >> memory: 128M > >>> > >>>> > > > >> > >>> > >>>> > > > >> Prometheus.yml > >>> > >>>> > > > >> > >>> > >>>> > > > >> scrape_configs: > >>> > >>>> > > > >> - job_name: 'docker' > >>> > >>>> > > > >> dockerswarm_sd_configs: > >>> > >>>> > > > >> - host: unix:///var/run/docker.sock > >>> > >>>> > > > >> role: nodes > >>> > >>>> > > > >> > >>> > >>>> > > > >> Error: > >>> > >>>> > > > >> [email protected] > >>> > >>>> <javascript:> | > >>> > >>>> > > level=error > >>> > >>>> > > > >> ts=2020-08-06T07:21:19.106Z caller=refresh.go:98 > >>> > >>>> component="discovery > >>> > >>>> > > > >> manager scrape" discovery=dockerswarm msg="Unable to > >>> refresh > >>> > >>>> target > >>> > >>>> > > groups" > >>> > >>>> > > > >> err="error while listing swarm nodes: Got permission > >>> denied > >>> > >>>> while > >>> > >>>> > > trying to > >>> > >>>> > > > >> connect to the Docker daemon socket at > >>> > >>>> unix:///var/run/docker.sock: > >>> > >>>> > > Get > >>> > >>>> > > > >> \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\": dial > >>> unix > >>> > >>>> > > > >> /var/run/docker.sock: connect: permission denied > >>> > >>>> > > > >> > >>> > >>>> > > > > > >>> > >>>> > > > > >>> > >>>> > > > -- > >>> > >>>> > > > You received this message because you are subscribed to the > >>> > >>>> Google > >>> > >>>> > > Groups "Prometheus Users" group. > >>> > >>>> > > > To unsubscribe from this group and stop receiving emails > >>> from it, > >>> > >>>> send > >>> > >>>> > > an email to [email protected] <javascript:>. > >>> > >>>> > > > To view this discussion on the web visit > >>> > >>>> > > > >>> > >>>> > >>> https://groups.google.com/d/msgid/prometheus-users/e5e55a73-7cc1-4c0c-99e3-0a09270df62bo%40googlegroups.com. > >>> > >>> > >>> > >>>> > >>> > >>>> > > > >>> > >>>> > > > >>> > >>>> > > > >>> > >>>> > > -- > >>> > >>>> > > Julien Pivotto > >>> > >>>> > > @roidelapluie > >>> > >>>> > > > >>> > >>>> > > >>> > >>>> > -- > >>> > >>>> > You received this message because you are subscribed to the > >>> Google > >>> > >>>> Groups "Prometheus Users" group. > >>> > >>>> > To unsubscribe from this group and stop receiving emails from > >>> it, > >>> > >>>> send an email to [email protected]. > >>> > >>>> > To view this discussion on the web visit > >>> > >>>> > >>> https://groups.google.com/d/msgid/prometheus-users/e5614621-f57a-466e-befd-269bf77d69c8o%40googlegroups.com. > >>> > >>> > >>> > >>>> > >>> > >>>> > >>> > >>>> > >>> > >>>> -- > >>> > >>>> Julien Pivotto > >>> > >>>> @roidelapluie > >>> > >>>> > >>> > >>> > >>> > > >>> > -- > >>> > You received this message because you are subscribed to the Google > >>> Groups "Prometheus Users" group. > >>> > To unsubscribe from this group and stop receiving emails from it, send > >>> an email to [email protected]. > >>> > To view this discussion on the web visit > >>> https://groups.google.com/d/msgid/prometheus-users/e058c64f-3db4-45c2-9550-c8db557d2a2cn%40googlegroups.com. > >>> > >>> > >>> > >>> > >>> -- > >>> Julien Pivotto > >>> @roidelapluie > >>> > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Prometheus Users" group. > >> To unsubscribe from this group and stop receiving emails from it, send an > >> email to [email protected]. > >> > > To view this discussion on the web visit > >> https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com > >> > >> <https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com?utm_medium=email&utm_source=footer> > >> . > >> > > > > > > -- > > Alexandru Duzsardi, > > *DevOps Engineer* > > *Skype:* alexinno83 > > *GPG/PGP Key*: https://keybase.io/aduzsardi/pgp_keys.asc > > *GitLab:* https://gitlab.com/aduzsardi > > *GitHub:* https://github.com/aduzsardi > > *LinkedIn:* https://www.linkedin.com/in/aduzsardi > > *E-mail:* [email protected] > > > > InFinIT Partners, > > *Address:* Str. Macinului Nr. 17, Cluj-Napoca, Romania > > *Web:* www.infinitpartners.com > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/f1962c97-b545-47e8-9877-3482bdfaadean%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20201116004417.GA544420%40oxygen.
