@julien ... Here: https://github.com/prometheus/prometheus/issues/8185
Let me know if you need more details On Monday, November 16, 2020 at 1:44:27 AM UTC+1 Julien Pivotto wrote: > On 15 Nov 16:30, Carlos Colaço wrote: > > Aight .. that fixed it for me too, was about to test it when i decided > to > > check in here first, so you were just faster :p > > > > > > If prometheus should or not run as root ... I am not sure either ... I > > think its a common practice to run stuff as root inside the containers > ... > > Cadvisor seems to be running as root ... but i am not entirely sure on > > this one, so take my words with a grain of salt. > > > > It should be however ... at least documented with a warning ... its > quite > > late here already but i can do it early in the morning tomorrow ... if > any > > of you has the chance in the meantime to try and verify this ... there > are > > some more tests that come to my mind ... > > > > Could it be that docker is not letting "nobody" read the sock? Maybe > Trying > > running Prometheus as another user instead of nobody or root? > > > > If nobody tries this i can try it tomorrow and maybe open PR to > > documentation with info about this. > > I guess it all depends on your distribution and how you run docker. Can > you explain more your setup? > > > > > > Kind regards. > > > > On Sunday, November 15, 2020 at 11:53:50 PM UTC+1 > > [email protected] wrote: > > > > > This worked for me , although i'm not sure we should be running > prometheus > > > as root > > > > > > version: '3.7' > > > > > > services: > > > prometheus: > > > image: prom/prometheus:v2.21.0 > > > > > > command: > > > - '--config.file=/etc/prometheus/prometheus.yml' > > > - '--storage.tsdb.path=/prometheus' > > > - '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-48h}' > > > user: root > > > volumes: > > > - /var/run/docker.sock:/var/run/docker.sock:ro > > > - ./prometheus.yml:/etc/prometheus/prometheus.yml > > > ports: > > > - target: 9090 > > > published: 9090 > > > mode: ingress > > > deploy: > > > labels: > > > - prometheus-job=prometheus > > > > > > mode: replicated > > > replicas: 1 > > > resources: > > > limits: > > > memory: 2048M > > > reservations: > > > memory: 512M > > > > > > On Sun, Nov 15, 2020 at 11:52 PM Kimo <[email protected]> wrote: > > > > > >> Hello, > > >> I've been facing the exact same issue today and its driving me > equally > > >> crazy. I tried running prometheus as root but still: > > >> > > >> level=error ts=2020-11-15T21:45:35.983Z caller=refresh.go:98 > > >> component="discovery manager scrape" discovery=dockerswarm > msg="Unable to > > >> refresh target groups" err="error while listing swarm *services*: > Cannot > > >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the > docker > > >> daemon running?" > > >> level=error ts=2020-11-15T21:45:35.984Z caller=refresh.go:98 > > >> component="discovery manager scrape" discovery=dockerswarm > msg="Unable to > > >> refresh target groups" err="error while listing swarm *nodes*: Cannot > > >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the > docker > > >> daemon running?" > > >> > > >> I think I've exhausted all the options I could try by myself and > would > > >> gladly appreciate any help at this point. > > >> Le dimanche 15 novembre 2020 à 21:54:26 UTC+1, Julien Pivotto a écrit > : > > >> > > >>> Can you run prometheus as nobody:docker? > > >>> On 15 Nov 12:23, Carlos Colaço wrote: > > >>> > sorry .. also tried changing the permissions which changed > nothing... > > >>> > > > >>> > ``` > > >>> > # chmod +r /var/run/docker.sock > > >>> > # ls -la /var/run/docker.sock > > >>> > srw-rw-r--. 1 root docker 0 Nov 15 20:12 /var/run/docker.sock > > >>> > # docker service update --force monitor_private > > >>> > ``` > > >>> > > > >>> > On Sunday, November 15, 2020 at 9:20:07 PM UTC+1 Carlos Colaço > wrote: > > >>> > > > >>> > > Hi all .. Having the same issue... > > >>> > > > > >>> > > https://github.com/prometheus/prometheus/issues/8185 > > >>> > > > > >>> > > > > >>> > > Also don't think changing permissions on docker sock is a good > > >>> option .. > > >>> > > that way you are giving permissions to anyone to access it and > that > > >>> is > > >>> > > something not desirable ... > > >>> > > > > >>> > > What i also tried to do instead ... since prometheus runs as > Nobody > > >>> ( uid: > > >>> > > 65534 ) ... i added it to the Docker group which changed nothing > =/ > > >>> > > > > >>> > > Any hints or solutions for this? driving me crazy trying > different > > >>> > > approaches and solutions.. nothing seems to work ... > > >>> > > > > >>> > > On Tuesday, August 11, 2020 at 7:03:12 AM UTC+2 > [email protected] > > >>> wrote: > > >>> > > > > >>> > >> Thanks Julien and Tom, > > >>> > >> > > >>> > >> I got the problem which i was facing, actually when we change > the > > >>> > >> permissions to read-write for docker.sock, permissions are only > > >>> changed > > >>> > >> till the docker daemon or docker service is restarted. Once the > > >>> > >> docker/daemon is restarted then the permissions for docker sock > > >>> changes > > >>> > >> back to the original one. > > >>> > >> > > >>> > >> Is there any way using which we can make permanent changes for > the > > >>> > >> permission of docker.sock or do we need to file a issue for the > > >>> same, as > > >>> > >> docker/daemon might be restarted for various reasons > > >>> > >> > > >>> > >> > > >>> > >> ? > > >>> > >> > > >>> > >> On Monday, 10 August 2020 12:40:17 UTC+5:30, Umang Goel wrote: > > >>> > >>> > > >>> > >>> Hello Julien, > > >>> > >>> > > >>> > >>> group_add is not allowed in docker swarm. Do you have any > other > > >>> > >>> workaround for this? > > >>> > >>> > > >>> > >>> -- > > >>> > >>> Umang > > >>> > >>> > > >>> > >>> On Monday, 10 August 2020 12:20:51 UTC+5:30, Julien Pivotto > wrote: > > >>> > >>>> > > >>> > >>>> > > >>> > >>>> Can you use: > > >>> > >>>> > > >>> > >>>> --group-add docker? > > >>> > >>>> > > >>> > >>>> or in compose v2 file: > > >>> > >>>> > > >>> > >>>> version: "2.4" > > >>> > >>>> services: > > >>> > >>>> prometheus: > > >>> > >>>> group_add: > > >>> > >>>> - docker > > >>> > >>>> > > >>> > >>>> > > >>> > >>>> On 09 Aug 22:48, Umang Goel wrote: > > >>> > >>>> > ls -l /var/run/docker.sock > > >>> > >>>> > > > >>> > >>>> > - srwxrw-rw- 1 root docker 0 Aug 7 11:31 > /var/run/docker.sock > > >>> > >>>> after > > >>> > >>>> > making changes as per Tom, > > >>> > >>>> > > > >>> > >>>> > On Sunday, 9 August 2020 02:16:28 UTC+5:30, Julien Pivotto > > >>> wrote: > > >>> > >>>> > > > > >>> > >>>> > > On 07 Aug 04:36, Umang Goel wrote: > > >>> > >>>> > > > Hello Tom, > > >>> > >>>> > > > > > >>> > >>>> > > > Even this is not working, I am still facing the same > issue. > > >>> Can > > >>> > >>>> you help > > >>> > >>>> > > me > > >>> > >>>> > > > how did you implement it. > > >>> > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > What are you current permissions on the > /var/run/docker.sock > > >>> ? > > >>> > >>>> > > > > >>> > >>>> > > ls -l /var/run/docker.sock > > >>> > >>>> > > > > >>> > >>>> > > > > > >>> > >>>> > > > On Friday, 7 August 2020 16:47:23 UTC+5:30, Tom Kun > wrote: > > >>> > >>>> > > > > > > >>> > >>>> > > > > Hello Umang, > > >>> > >>>> > > > > > > >>> > >>>> > > > > What are you current permissions on the > > >>> /var/run/docker.sock ? > > >>> > >>>> > > > > > > >>> > >>>> > > > > I faced the same issue, and to start and no rebuild > the > > >>> > >>>> Prometheus > > >>> > >>>> > > image > > >>> > >>>> > > > > with the appropriate user. > > >>> > >>>> > > > > I put the rights to read and write the docker.socket. > > >>> > >>>> > > > > > > >>> > >>>> > > > > sudo chmod 766 /var/run/docker.sock > > >>> > >>>> > > > > > > >>> > >>>> > > > > I hope this gonna help you. > > >>> > >>>> > > > > > > >>> > >>>> > > > > > > >>> > >>>> > > > > On Friday, 7 August 2020 11:59:32 UTC+2, Umang Goel > > >>> wrote: > > >>> > >>>> > > > >> > > >>> > >>>> > > > >> Hello Community, > > >>> > >>>> > > > >> > > >>> > >>>> > > > >> I tired using Docker Swarm Service Discovery in > > >>> prometheus, > > >>> > >>>> but > > >>> > >>>> > > facing > > >>> > >>>> > > > >> problems using it. I followed the docker swarm > support > > >>> > >>>> documentation > > >>> > >>>> > > > >> <https://prometheus.io/docs/guides/dockerswarm/>. > > >>> Created a > > >>> > >>>> > > daemon.json > > >>> > >>>> > > > >> file and mounted /var/run/docker.sock in prometheus > > >>> container. > > >>> > >>>> > > Container is > > >>> > >>>> > > > >> giving permission denied error as prometheus is > running > > >>> as > > >>> > >>>> nobody and > > >>> > >>>> > > > >> doesn't have access to mounted /var/run/docker.sock. > > >>> Below is > > >>> > >>>> my > > >>> > >>>> > > > >> prometheus.yml. > > >>> > >>>> > > > >> Prometheus Version : v2.20.1 > > >>> > >>>> > > > >> > > >>> > >>>> > > > >> prometheus: > > >>> > >>>> > > > >> image: prom/prometheus > > >>> > >>>> > > > >> networks: > > >>> > >>>> > > > >> - monitor > > >>> > >>>> > > > >> ports: > > >>> > >>>> > > > >> - "9090:9090" > > >>> > >>>> > > > >> command: > > >>> > >>>> > > > >> - '--config.file=/etc/prometheus/prometheus.yml' > > >>> > >>>> > > > >> - '--storage.tsdb.path=/prometheus' > > >>> > >>>> > > > >> - > > >>> > >>>> '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-24h}' > > >>> > >>>> > > > >> volumes: > > >>> > >>>> > > > >> - prometheus:/prometheus > > >>> > >>>> > > > >> - /home/efs/devops/dsm:/etc/prometheus:ro > > >>> > >>>> > > > >> - /var/run/docker.sock:/var/run/docker.sock:ro > > >>> > >>>> > > > >> deploy: > > >>> > >>>> > > > >> mode: replicated > > >>> > >>>> > > > >> replicas: 1 > > >>> > >>>> > > > >> resources: > > >>> > >>>> > > > >> limits: > > >>> > >>>> > > > >> memory: 1024M > > >>> > >>>> > > > >> reservations: > > >>> > >>>> > > > >> memory: 128M > > >>> > >>>> > > > >> > > >>> > >>>> > > > >> Prometheus.yml > > >>> > >>>> > > > >> > > >>> > >>>> > > > >> scrape_configs: > > >>> > >>>> > > > >> - job_name: 'docker' > > >>> > >>>> > > > >> dockerswarm_sd_configs: > > >>> > >>>> > > > >> - host: unix:///var/run/docker.sock > > >>> > >>>> > > > >> role: nodes > > >>> > >>>> > > > >> > > >>> > >>>> > > > >> Error: > > >>> > >>>> > > > >> [email protected] > > >>> > >>>> <javascript:> | > > >>> > >>>> > > level=error > > >>> > >>>> > > > >> ts=2020-08-06T07:21:19.106Z caller=refresh.go:98 > > >>> > >>>> component="discovery > > >>> > >>>> > > > >> manager scrape" discovery=dockerswarm msg="Unable to > > >>> refresh > > >>> > >>>> target > > >>> > >>>> > > groups" > > >>> > >>>> > > > >> err="error while listing swarm nodes: Got permission > > >>> denied > > >>> > >>>> while > > >>> > >>>> > > trying to > > >>> > >>>> > > > >> connect to the Docker daemon socket at > > >>> > >>>> unix:///var/run/docker.sock: > > >>> > >>>> > > Get > > >>> > >>>> > > > >> \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\": > dial > > >>> unix > > >>> > >>>> > > > >> /var/run/docker.sock: connect: permission denied > > >>> > >>>> > > > >> > > >>> > >>>> > > > > > > >>> > >>>> > > > > > >>> > >>>> > > > -- > > >>> > >>>> > > > You received this message because you are subscribed to > the > > >>> > >>>> Google > > >>> > >>>> > > Groups "Prometheus Users" group. > > >>> > >>>> > > > To unsubscribe from this group and stop receiving > emails > > >>> from it, > > >>> > >>>> send > > >>> > >>>> > > an email to [email protected] <javascript:>. > > >>> > >>>> > > > To view this discussion on the web visit > > >>> > >>>> > > > > >>> > >>>> > > >>> > https://groups.google.com/d/msgid/prometheus-users/e5e55a73-7cc1-4c0c-99e3-0a09270df62bo%40googlegroups.com. > > > > >>> > > >>> > >>>> > > >>> > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > > > >>> > >>>> > > -- > > >>> > >>>> > > Julien Pivotto > > >>> > >>>> > > @roidelapluie > > >>> > >>>> > > > > >>> > >>>> > > > >>> > >>>> > -- > > >>> > >>>> > You received this message because you are subscribed to the > > >>> Google > > >>> > >>>> Groups "Prometheus Users" group. > > >>> > >>>> > To unsubscribe from this group and stop receiving emails > from > > >>> it, > > >>> > >>>> send an email to [email protected]. > > >>> > >>>> > To view this discussion on the web visit > > >>> > >>>> > > >>> > https://groups.google.com/d/msgid/prometheus-users/e5614621-f57a-466e-befd-269bf77d69c8o%40googlegroups.com. > > > > >>> > > >>> > >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > >>>> -- > > >>> > >>>> Julien Pivotto > > >>> > >>>> @roidelapluie > > >>> > >>>> > > >>> > >>> > > >>> > > > >>> > -- > > >>> > You received this message because you are subscribed to the Google > > >>> Groups "Prometheus Users" group. > > >>> > To unsubscribe from this group and stop receiving emails from it, > send > > >>> an email to [email protected]. > > >>> > To view this discussion on the web visit > > >>> > https://groups.google.com/d/msgid/prometheus-users/e058c64f-3db4-45c2-9550-c8db557d2a2cn%40googlegroups.com. > > > > >>> > > >>> > > >>> > > >>> -- > > >>> Julien Pivotto > > >>> @roidelapluie > > >>> > > >> -- > > >> You received this message because you are subscribed to the Google > Groups > > >> "Prometheus Users" group. > > >> To unsubscribe from this group and stop receiving emails from it, > send an > > >> email to [email protected]. > > >> > > > To view this discussion on the web visit > > >> > https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com > > > >> < > https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com?utm_medium=email&utm_source=footer > > > > >> . > > >> > > > > > > > > > -- > > > Alexandru Duzsardi, > > > *DevOps Engineer* > > > *Skype:* alexinno83 > > > *GPG/PGP Key*: https://keybase.io/aduzsardi/pgp_keys.asc > > > *GitLab:* https://gitlab.com/aduzsardi > > > *GitHub:* https://github.com/aduzsardi > > > *LinkedIn:* https://www.linkedin.com/in/aduzsardi > > > *E-mail:* [email protected] > > > > > > InFinIT Partners, > > > *Address:* Str. Macinului Nr. 17, Cluj-Napoca, Romania > > > *Web:* www.infinitpartners.com > > > > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/f1962c97-b545-47e8-9877-3482bdfaadean%40googlegroups.com > . > > > -- > Julien Pivotto > @roidelapluie > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/84bf5f2a-d3fb-434f-9ff2-b9209a3f50bdn%40googlegroups.com.
