This worked for me , although i'm not sure we should be running prometheus
as root
version: '3.7'
services:
prometheus:
image: prom/prometheus:v2.21.0
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-48h}'
user: root
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- target: 9090
published: 9090
mode: ingress
deploy:
labels:
- prometheus-job=prometheus
mode: replicated
replicas: 1
resources:
limits:
memory: 2048M
reservations:
memory: 512M
On Sun, Nov 15, 2020 at 11:52 PM Kimo <[email protected]> wrote:
> Hello,
> I've been facing the exact same issue today and its driving me equally
> crazy. I tried running prometheus as root but still:
>
> level=error ts=2020-11-15T21:45:35.983Z caller=refresh.go:98
> component="discovery manager scrape" discovery=dockerswarm msg="Unable to
> refresh target groups" err="error while listing swarm *services*: Cannot
> connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker
> daemon running?"
> level=error ts=2020-11-15T21:45:35.984Z caller=refresh.go:98
> component="discovery manager scrape" discovery=dockerswarm msg="Unable to
> refresh target groups" err="error while listing swarm *nodes*: Cannot
> connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker
> daemon running?"
>
> I think I've exhausted all the options I could try by myself and would
> gladly appreciate any help at this point.
> Le dimanche 15 novembre 2020 à 21:54:26 UTC+1, Julien Pivotto a écrit :
>
>> Can you run prometheus as nobody:docker?
>> On 15 Nov 12:23, Carlos Colaço wrote:
>> > sorry .. also tried changing the permissions which changed nothing...
>> >
>> > ```
>> > # chmod +r /var/run/docker.sock
>> > # ls -la /var/run/docker.sock
>> > srw-rw-r--. 1 root docker 0 Nov 15 20:12 /var/run/docker.sock
>> > # docker service update --force monitor_private
>> > ```
>> >
>> > On Sunday, November 15, 2020 at 9:20:07 PM UTC+1 Carlos Colaço wrote:
>> >
>> > > Hi all .. Having the same issue...
>> > >
>> > > https://github.com/prometheus/prometheus/issues/8185
>> > >
>> > >
>> > > Also don't think changing permissions on docker sock is a good option
>> ..
>> > > that way you are giving permissions to anyone to access it and that
>> is
>> > > something not desirable ...
>> > >
>> > > What i also tried to do instead ... since prometheus runs as Nobody (
>> uid:
>> > > 65534 ) ... i added it to the Docker group which changed nothing =/
>> > >
>> > > Any hints or solutions for this? driving me crazy trying different
>> > > approaches and solutions.. nothing seems to work ...
>> > >
>> > > On Tuesday, August 11, 2020 at 7:03:12 AM UTC+2 [email protected]
>> wrote:
>> > >
>> > >> Thanks Julien and Tom,
>> > >>
>> > >> I got the problem which i was facing, actually when we change the
>> > >> permissions to read-write for docker.sock, permissions are only
>> changed
>> > >> till the docker daemon or docker service is restarted. Once the
>> > >> docker/daemon is restarted then the permissions for docker sock
>> changes
>> > >> back to the original one.
>> > >>
>> > >> Is there any way using which we can make permanent changes for the
>> > >> permission of docker.sock or do we need to file a issue for the
>> same, as
>> > >> docker/daemon might be restarted for various reasons
>> > >>
>> > >>
>> > >> ?
>> > >>
>> > >> On Monday, 10 August 2020 12:40:17 UTC+5:30, Umang Goel wrote:
>> > >>>
>> > >>> Hello Julien,
>> > >>>
>> > >>> group_add is not allowed in docker swarm. Do you have any other
>> > >>> workaround for this?
>> > >>>
>> > >>> --
>> > >>> Umang
>> > >>>
>> > >>> On Monday, 10 August 2020 12:20:51 UTC+5:30, Julien Pivotto wrote:
>> > >>>>
>> > >>>>
>> > >>>> Can you use:
>> > >>>>
>> > >>>> --group-add docker?
>> > >>>>
>> > >>>> or in compose v2 file:
>> > >>>>
>> > >>>> version: "2.4"
>> > >>>> services:
>> > >>>> prometheus:
>> > >>>> group_add:
>> > >>>> - docker
>> > >>>>
>> > >>>>
>> > >>>> On 09 Aug 22:48, Umang Goel wrote:
>> > >>>> > ls -l /var/run/docker.sock
>> > >>>> >
>> > >>>> > - srwxrw-rw- 1 root docker 0 Aug 7 11:31 /var/run/docker.sock
>> > >>>> after
>> > >>>> > making changes as per Tom,
>> > >>>> >
>> > >>>> > On Sunday, 9 August 2020 02:16:28 UTC+5:30, Julien Pivotto
>> wrote:
>> > >>>> > >
>> > >>>> > > On 07 Aug 04:36, Umang Goel wrote:
>> > >>>> > > > Hello Tom,
>> > >>>> > > >
>> > >>>> > > > Even this is not working, I am still facing the same issue.
>> Can
>> > >>>> you help
>> > >>>> > > me
>> > >>>> > > > how did you implement it.
>> > >>>> > >
>> > >>>> > >
>> > >>>> > > What are you current permissions on the /var/run/docker.sock ?
>> > >>>> > >
>> > >>>> > > ls -l /var/run/docker.sock
>> > >>>> > >
>> > >>>> > > >
>> > >>>> > > > On Friday, 7 August 2020 16:47:23 UTC+5:30, Tom Kun wrote:
>> > >>>> > > > >
>> > >>>> > > > > Hello Umang,
>> > >>>> > > > >
>> > >>>> > > > > What are you current permissions on the
>> /var/run/docker.sock ?
>> > >>>> > > > >
>> > >>>> > > > > I faced the same issue, and to start and no rebuild the
>> > >>>> Prometheus
>> > >>>> > > image
>> > >>>> > > > > with the appropriate user.
>> > >>>> > > > > I put the rights to read and write the docker.socket.
>> > >>>> > > > >
>> > >>>> > > > > sudo chmod 766 /var/run/docker.sock
>> > >>>> > > > >
>> > >>>> > > > > I hope this gonna help you.
>> > >>>> > > > >
>> > >>>> > > > >
>> > >>>> > > > > On Friday, 7 August 2020 11:59:32 UTC+2, Umang Goel wrote:
>> > >>>> > > > >>
>> > >>>> > > > >> Hello Community,
>> > >>>> > > > >>
>> > >>>> > > > >> I tired using Docker Swarm Service Discovery in
>> prometheus,
>> > >>>> but
>> > >>>> > > facing
>> > >>>> > > > >> problems using it. I followed the docker swarm support
>> > >>>> documentation
>> > >>>> > > > >> <https://prometheus.io/docs/guides/dockerswarm/>.
>> Created a
>> > >>>> > > daemon.json
>> > >>>> > > > >> file and mounted /var/run/docker.sock in prometheus
>> container.
>> > >>>> > > Container is
>> > >>>> > > > >> giving permission denied error as prometheus is running
>> as
>> > >>>> nobody and
>> > >>>> > > > >> doesn't have access to mounted /var/run/docker.sock.
>> Below is
>> > >>>> my
>> > >>>> > > > >> prometheus.yml.
>> > >>>> > > > >> Prometheus Version : v2.20.1
>> > >>>> > > > >>
>> > >>>> > > > >> prometheus:
>> > >>>> > > > >> image: prom/prometheus
>> > >>>> > > > >> networks:
>> > >>>> > > > >> - monitor
>> > >>>> > > > >> ports:
>> > >>>> > > > >> - "9090:9090"
>> > >>>> > > > >> command:
>> > >>>> > > > >> - '--config.file=/etc/prometheus/prometheus.yml'
>> > >>>> > > > >> - '--storage.tsdb.path=/prometheus'
>> > >>>> > > > >> -
>> > >>>> '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-24h}'
>> > >>>> > > > >> volumes:
>> > >>>> > > > >> - prometheus:/prometheus
>> > >>>> > > > >> - /home/efs/devops/dsm:/etc/prometheus:ro
>> > >>>> > > > >> - /var/run/docker.sock:/var/run/docker.sock:ro
>> > >>>> > > > >> deploy:
>> > >>>> > > > >> mode: replicated
>> > >>>> > > > >> replicas: 1
>> > >>>> > > > >> resources:
>> > >>>> > > > >> limits:
>> > >>>> > > > >> memory: 1024M
>> > >>>> > > > >> reservations:
>> > >>>> > > > >> memory: 128M
>> > >>>> > > > >>
>> > >>>> > > > >> Prometheus.yml
>> > >>>> > > > >>
>> > >>>> > > > >> scrape_configs:
>> > >>>> > > > >> - job_name: 'docker'
>> > >>>> > > > >> dockerswarm_sd_configs:
>> > >>>> > > > >> - host: unix:///var/run/docker.sock
>> > >>>> > > > >> role: nodes
>> > >>>> > > > >>
>> > >>>> > > > >> Error:
>> > >>>> > > > >> [email protected]
>> > >>>> <javascript:> |
>> > >>>> > > level=error
>> > >>>> > > > >> ts=2020-08-06T07:21:19.106Z caller=refresh.go:98
>> > >>>> component="discovery
>> > >>>> > > > >> manager scrape" discovery=dockerswarm msg="Unable to
>> refresh
>> > >>>> target
>> > >>>> > > groups"
>> > >>>> > > > >> err="error while listing swarm nodes: Got permission
>> denied
>> > >>>> while
>> > >>>> > > trying to
>> > >>>> > > > >> connect to the Docker daemon socket at
>> > >>>> unix:///var/run/docker.sock:
>> > >>>> > > Get
>> > >>>> > > > >> \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\": dial
>> unix
>> > >>>> > > > >> /var/run/docker.sock: connect: permission denied
>> > >>>> > > > >>
>> > >>>> > > > >
>> > >>>> > > >
>> > >>>> > > > --
>> > >>>> > > > You received this message because you are subscribed to the
>> > >>>> Google
>> > >>>> > > Groups "Prometheus Users" group.
>> > >>>> > > > To unsubscribe from this group and stop receiving emails
>> from it,
>> > >>>> send
>> > >>>> > > an email to [email protected] <javascript:>.
>> > >>>> > > > To view this discussion on the web visit
>> > >>>> > >
>> > >>>>
>> https://groups.google.com/d/msgid/prometheus-users/e5e55a73-7cc1-4c0c-99e3-0a09270df62bo%40googlegroups.com.
>>
>> > >>>>
>> > >>>> > >
>> > >>>> > >
>> > >>>> > >
>> > >>>> > > --
>> > >>>> > > Julien Pivotto
>> > >>>> > > @roidelapluie
>> > >>>> > >
>> > >>>> >
>> > >>>> > --
>> > >>>> > You received this message because you are subscribed to the
>> Google
>> > >>>> Groups "Prometheus Users" group.
>> > >>>> > To unsubscribe from this group and stop receiving emails from
>> it,
>> > >>>> send an email to [email protected].
>> > >>>> > To view this discussion on the web visit
>> > >>>>
>> https://groups.google.com/d/msgid/prometheus-users/e5614621-f57a-466e-befd-269bf77d69c8o%40googlegroups.com.
>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> --
>> > >>>> Julien Pivotto
>> > >>>> @roidelapluie
>> > >>>>
>> > >>>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to [email protected].
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/e058c64f-3db4-45c2-9550-c8db557d2a2cn%40googlegroups.com.
>>
>>
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Alexandru Duzsardi,
*DevOps Engineer*
*Skype:* alexinno83
*GPG/PGP Key*: https://keybase.io/aduzsardi/pgp_keys.asc
*GitLab:* https://gitlab.com/aduzsardi
*GitHub:* https://github.com/aduzsardi
*LinkedIn:* https://www.linkedin.com/in/aduzsardi
*E-mail:* [email protected]
InFinIT Partners,
*Address:* Str. Macinului Nr. 17, Cluj-Napoca, Romania
*Web:* www.infinitpartners.com
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAAa91wo%2BrVJNOHZjTusEQX24jNQg05DUpL1Y6t2vthw%2Bsn2f4A%40mail.gmail.com.
