Hi all ...
as promised ... built a new image ... and ran prometheus as another user
...
Dockerfile:
```
FROM prom/prometheus
USER root
RUN addgroup -g 1000 prometheus
RUN adduser -D -H -u 1000 -G prometheus -s /bin/nologin prometheus
USER prometheus
```
Docker compose file:
```
version: '3.3'
services:
private:
image: 4s3ti/prometheus-test
ports:
- 9090:9090
networks:
- dockadmin_rp
- private
volumes:
- /srv/data/prometheus/config:/etc/prometheus
- /srv/data/prometheus/test-data:/prometheus
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
replicas: 1
restart_policy:
condition: on-failure
placement:
constraints:
- node.labels.type == private
networks:
private:
dockadmin_rp:
external: true
```
and the issue still persists, after some googling ... Apparently for a
process to have access to the docker.sock withing the container, it needs
to run as root, or one should use something as docker-socket-proxy.
If Prometheus team don't want to change the default way Prometheus runs
inside a docker container, which i completely understand, a note about this
should be added on the https://prometheus.io/docs/guides/dockerswarm/ page
On Monday, November 16, 2020 at 2:12:48 AM UTC+1 Carlos Colaço wrote:
> @julien ... Here:
>
> https://github.com/prometheus/prometheus/issues/8185
>
> Let me know if you need more details
>
> On Monday, November 16, 2020 at 1:44:27 AM UTC+1 Julien Pivotto wrote:
>
>> On 15 Nov 16:30, Carlos Colaço wrote:
>> > Aight .. that fixed it for me too, was about to test it when i decided
>> to
>> > check in here first, so you were just faster :p
>> >
>> >
>> > If prometheus should or not run as root ... I am not sure either ... I
>> > think its a common practice to run stuff as root inside the containers
>> ...
>> > Cadvisor seems to be running as root ... but i am not entirely sure on
>> > this one, so take my words with a grain of salt.
>> >
>> > It should be however ... at least documented with a warning ... its
>> quite
>> > late here already but i can do it early in the morning tomorrow ... if
>> any
>> > of you has the chance in the meantime to try and verify this ... there
>> are
>> > some more tests that come to my mind ...
>> >
>> > Could it be that docker is not letting "nobody" read the sock? Maybe
>> Trying
>> > running Prometheus as another user instead of nobody or root?
>> >
>> > If nobody tries this i can try it tomorrow and maybe open PR to
>> > documentation with info about this.
>>
>> I guess it all depends on your distribution and how you run docker. Can
>> you explain more your setup?
>>
>>
>> >
>> > Kind regards.
>> >
>> > On Sunday, November 15, 2020 at 11:53:50 PM UTC+1
>> > [email protected] wrote:
>> >
>> > > This worked for me , although i'm not sure we should be running
>> prometheus
>> > > as root
>> > >
>> > > version: '3.7'
>> > >
>> > > services:
>> > > prometheus:
>> > > image: prom/prometheus:v2.21.0
>> > >
>> > > command:
>> > > - '--config.file=/etc/prometheus/prometheus.yml'
>> > > - '--storage.tsdb.path=/prometheus'
>> > > - '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-48h}'
>> > > user: root
>> > > volumes:
>> > > - /var/run/docker.sock:/var/run/docker.sock:ro
>> > > - ./prometheus.yml:/etc/prometheus/prometheus.yml
>> > > ports:
>> > > - target: 9090
>> > > published: 9090
>> > > mode: ingress
>> > > deploy:
>> > > labels:
>> > > - prometheus-job=prometheus
>> > >
>> > > mode: replicated
>> > > replicas: 1
>> > > resources:
>> > > limits:
>> > > memory: 2048M
>> > > reservations:
>> > > memory: 512M
>> > >
>> > > On Sun, Nov 15, 2020 at 11:52 PM Kimo <[email protected]> wrote:
>> > >
>> > >> Hello,
>> > >> I've been facing the exact same issue today and its driving me
>> equally
>> > >> crazy. I tried running prometheus as root but still:
>> > >>
>> > >> level=error ts=2020-11-15T21:45:35.983Z caller=refresh.go:98
>> > >> component="discovery manager scrape" discovery=dockerswarm
>> msg="Unable to
>> > >> refresh target groups" err="error while listing swarm *services*:
>> Cannot
>> > >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the
>> docker
>> > >> daemon running?"
>> > >> level=error ts=2020-11-15T21:45:35.984Z caller=refresh.go:98
>> > >> component="discovery manager scrape" discovery=dockerswarm
>> msg="Unable to
>> > >> refresh target groups" err="error while listing swarm *nodes*:
>> Cannot
>> > >> connect to the Docker daemon at unix:///var/run/docker.sock. Is the
>> docker
>> > >> daemon running?"
>> > >>
>> > >> I think I've exhausted all the options I could try by myself and
>> would
>> > >> gladly appreciate any help at this point.
>> > >> Le dimanche 15 novembre 2020 à 21:54:26 UTC+1, Julien Pivotto a
>> écrit :
>> > >>
>> > >>> Can you run prometheus as nobody:docker?
>> > >>> On 15 Nov 12:23, Carlos Colaço wrote:
>> > >>> > sorry .. also tried changing the permissions which changed
>> nothing...
>> > >>> >
>> > >>> > ```
>> > >>> > # chmod +r /var/run/docker.sock
>> > >>> > # ls -la /var/run/docker.sock
>> > >>> > srw-rw-r--. 1 root docker 0 Nov 15 20:12 /var/run/docker.sock
>> > >>> > # docker service update --force monitor_private
>> > >>> > ```
>> > >>> >
>> > >>> > On Sunday, November 15, 2020 at 9:20:07 PM UTC+1 Carlos Colaço
>> wrote:
>> > >>> >
>> > >>> > > Hi all .. Having the same issue...
>> > >>> > >
>> > >>> > > https://github.com/prometheus/prometheus/issues/8185
>> > >>> > >
>> > >>> > >
>> > >>> > > Also don't think changing permissions on docker sock is a good
>> > >>> option ..
>> > >>> > > that way you are giving permissions to anyone to access it and
>> that
>> > >>> is
>> > >>> > > something not desirable ...
>> > >>> > >
>> > >>> > > What i also tried to do instead ... since prometheus runs as
>> Nobody
>> > >>> ( uid:
>> > >>> > > 65534 ) ... i added it to the Docker group which changed
>> nothing =/
>> > >>> > >
>> > >>> > > Any hints or solutions for this? driving me crazy trying
>> different
>> > >>> > > approaches and solutions.. nothing seems to work ...
>> > >>> > >
>> > >>> > > On Tuesday, August 11, 2020 at 7:03:12 AM UTC+2
>> [email protected]
>> > >>> wrote:
>> > >>> > >
>> > >>> > >> Thanks Julien and Tom,
>> > >>> > >>
>> > >>> > >> I got the problem which i was facing, actually when we change
>> the
>> > >>> > >> permissions to read-write for docker.sock, permissions are
>> only
>> > >>> changed
>> > >>> > >> till the docker daemon or docker service is restarted. Once
>> the
>> > >>> > >> docker/daemon is restarted then the permissions for docker
>> sock
>> > >>> changes
>> > >>> > >> back to the original one.
>> > >>> > >>
>> > >>> > >> Is there any way using which we can make permanent changes for
>> the
>> > >>> > >> permission of docker.sock or do we need to file a issue for
>> the
>> > >>> same, as
>> > >>> > >> docker/daemon might be restarted for various reasons
>> > >>> > >>
>> > >>> > >>
>> > >>> > >> ?
>> > >>> > >>
>> > >>> > >> On Monday, 10 August 2020 12:40:17 UTC+5:30, Umang Goel wrote:
>> > >>> > >>>
>> > >>> > >>> Hello Julien,
>> > >>> > >>>
>> > >>> > >>> group_add is not allowed in docker swarm. Do you have any
>> other
>> > >>> > >>> workaround for this?
>> > >>> > >>>
>> > >>> > >>> --
>> > >>> > >>> Umang
>> > >>> > >>>
>> > >>> > >>> On Monday, 10 August 2020 12:20:51 UTC+5:30, Julien Pivotto
>> wrote:
>> > >>> > >>>>
>> > >>> > >>>>
>> > >>> > >>>> Can you use:
>> > >>> > >>>>
>> > >>> > >>>> --group-add docker?
>> > >>> > >>>>
>> > >>> > >>>> or in compose v2 file:
>> > >>> > >>>>
>> > >>> > >>>> version: "2.4"
>> > >>> > >>>> services:
>> > >>> > >>>> prometheus:
>> > >>> > >>>> group_add:
>> > >>> > >>>> - docker
>> > >>> > >>>>
>> > >>> > >>>>
>> > >>> > >>>> On 09 Aug 22:48, Umang Goel wrote:
>> > >>> > >>>> > ls -l /var/run/docker.sock
>> > >>> > >>>> >
>> > >>> > >>>> > - srwxrw-rw- 1 root docker 0 Aug 7 11:31
>> /var/run/docker.sock
>> > >>> > >>>> after
>> > >>> > >>>> > making changes as per Tom,
>> > >>> > >>>> >
>> > >>> > >>>> > On Sunday, 9 August 2020 02:16:28 UTC+5:30, Julien Pivotto
>> > >>> wrote:
>> > >>> > >>>> > >
>> > >>> > >>>> > > On 07 Aug 04:36, Umang Goel wrote:
>> > >>> > >>>> > > > Hello Tom,
>> > >>> > >>>> > > >
>> > >>> > >>>> > > > Even this is not working, I am still facing the same
>> issue.
>> > >>> Can
>> > >>> > >>>> you help
>> > >>> > >>>> > > me
>> > >>> > >>>> > > > how did you implement it.
>> > >>> > >>>> > >
>> > >>> > >>>> > >
>> > >>> > >>>> > > What are you current permissions on the
>> /var/run/docker.sock
>> > >>> ?
>> > >>> > >>>> > >
>> > >>> > >>>> > > ls -l /var/run/docker.sock
>> > >>> > >>>> > >
>> > >>> > >>>> > > >
>> > >>> > >>>> > > > On Friday, 7 August 2020 16:47:23 UTC+5:30, Tom Kun
>> wrote:
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > > > Hello Umang,
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > > > What are you current permissions on the
>> > >>> /var/run/docker.sock ?
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > > > I faced the same issue, and to start and no rebuild
>> the
>> > >>> > >>>> Prometheus
>> > >>> > >>>> > > image
>> > >>> > >>>> > > > > with the appropriate user.
>> > >>> > >>>> > > > > I put the rights to read and write the
>> docker.socket.
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > > > sudo chmod 766 /var/run/docker.sock
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > > > I hope this gonna help you.
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > > > On Friday, 7 August 2020 11:59:32 UTC+2, Umang Goel
>> > >>> wrote:
>> > >>> > >>>> > > > >>
>> > >>> > >>>> > > > >> Hello Community,
>> > >>> > >>>> > > > >>
>> > >>> > >>>> > > > >> I tired using Docker Swarm Service Discovery in
>> > >>> prometheus,
>> > >>> > >>>> but
>> > >>> > >>>> > > facing
>> > >>> > >>>> > > > >> problems using it. I followed the docker swarm
>> support
>> > >>> > >>>> documentation
>> > >>> > >>>> > > > >> <https://prometheus.io/docs/guides/dockerswarm/>.
>> > >>> Created a
>> > >>> > >>>> > > daemon.json
>> > >>> > >>>> > > > >> file and mounted /var/run/docker.sock in prometheus
>> > >>> container.
>> > >>> > >>>> > > Container is
>> > >>> > >>>> > > > >> giving permission denied error as prometheus is
>> running
>> > >>> as
>> > >>> > >>>> nobody and
>> > >>> > >>>> > > > >> doesn't have access to mounted
>> /var/run/docker.sock.
>> > >>> Below is
>> > >>> > >>>> my
>> > >>> > >>>> > > > >> prometheus.yml.
>> > >>> > >>>> > > > >> Prometheus Version : v2.20.1
>> > >>> > >>>> > > > >>
>> > >>> > >>>> > > > >> prometheus:
>> > >>> > >>>> > > > >> image: prom/prometheus
>> > >>> > >>>> > > > >> networks:
>> > >>> > >>>> > > > >> - monitor
>> > >>> > >>>> > > > >> ports:
>> > >>> > >>>> > > > >> - "9090:9090"
>> > >>> > >>>> > > > >> command:
>> > >>> > >>>> > > > >> - '--config.file=/etc/prometheus/prometheus.yml'
>> > >>> > >>>> > > > >> - '--storage.tsdb.path=/prometheus'
>> > >>> > >>>> > > > >> -
>> > >>> > >>>> '--storage.tsdb.retention=${PROMETHEUS_RETENTION:-24h}'
>> > >>> > >>>> > > > >> volumes:
>> > >>> > >>>> > > > >> - prometheus:/prometheus
>> > >>> > >>>> > > > >> - /home/efs/devops/dsm:/etc/prometheus:ro
>> > >>> > >>>> > > > >> - /var/run/docker.sock:/var/run/docker.sock:ro
>> > >>> > >>>> > > > >> deploy:
>> > >>> > >>>> > > > >> mode: replicated
>> > >>> > >>>> > > > >> replicas: 1
>> > >>> > >>>> > > > >> resources:
>> > >>> > >>>> > > > >> limits:
>> > >>> > >>>> > > > >> memory: 1024M
>> > >>> > >>>> > > > >> reservations:
>> > >>> > >>>> > > > >> memory: 128M
>> > >>> > >>>> > > > >>
>> > >>> > >>>> > > > >> Prometheus.yml
>> > >>> > >>>> > > > >>
>> > >>> > >>>> > > > >> scrape_configs:
>> > >>> > >>>> > > > >> - job_name: 'docker'
>> > >>> > >>>> > > > >> dockerswarm_sd_configs:
>> > >>> > >>>> > > > >> - host: unix:///var/run/docker.sock
>> > >>> > >>>> > > > >> role: nodes
>> > >>> > >>>> > > > >>
>> > >>> > >>>> > > > >> Error:
>> > >>> > >>>> > > > >> [email protected]
>> > >>> > >>>> <javascript:> |
>> > >>> > >>>> > > level=error
>> > >>> > >>>> > > > >> ts=2020-08-06T07:21:19.106Z caller=refresh.go:98
>> > >>> > >>>> component="discovery
>> > >>> > >>>> > > > >> manager scrape" discovery=dockerswarm msg="Unable
>> to
>> > >>> refresh
>> > >>> > >>>> target
>> > >>> > >>>> > > groups"
>> > >>> > >>>> > > > >> err="error while listing swarm nodes: Got
>> permission
>> > >>> denied
>> > >>> > >>>> while
>> > >>> > >>>> > > trying to
>> > >>> > >>>> > > > >> connect to the Docker daemon socket at
>> > >>> > >>>> unix:///var/run/docker.sock:
>> > >>> > >>>> > > Get
>> > >>> > >>>> > > > >> \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/nodes\":
>> dial
>> > >>> unix
>> > >>> > >>>> > > > >> /var/run/docker.sock: connect: permission denied
>> > >>> > >>>> > > > >>
>> > >>> > >>>> > > > >
>> > >>> > >>>> > > >
>> > >>> > >>>> > > > --
>> > >>> > >>>> > > > You received this message because you are subscribed
>> to the
>> > >>> > >>>> Google
>> > >>> > >>>> > > Groups "Prometheus Users" group.
>> > >>> > >>>> > > > To unsubscribe from this group and stop receiving
>> emails
>> > >>> from it,
>> > >>> > >>>> send
>> > >>> > >>>> > > an email to [email protected] <javascript:>.
>> > >>> > >>>> > > > To view this discussion on the web visit
>> > >>> > >>>> > >
>> > >>> > >>>>
>> > >>>
>> https://groups.google.com/d/msgid/prometheus-users/e5e55a73-7cc1-4c0c-99e3-0a09270df62bo%40googlegroups.com.
>>
>>
>> > >>>
>> > >>> > >>>>
>> > >>> > >>>> > >
>> > >>> > >>>> > >
>> > >>> > >>>> > >
>> > >>> > >>>> > > --
>> > >>> > >>>> > > Julien Pivotto
>> > >>> > >>>> > > @roidelapluie
>> > >>> > >>>> > >
>> > >>> > >>>> >
>> > >>> > >>>> > --
>> > >>> > >>>> > You received this message because you are subscribed to
>> the
>> > >>> Google
>> > >>> > >>>> Groups "Prometheus Users" group.
>> > >>> > >>>> > To unsubscribe from this group and stop receiving emails
>> from
>> > >>> it,
>> > >>> > >>>> send an email to [email protected].
>> > >>> > >>>> > To view this discussion on the web visit
>> > >>> > >>>>
>> > >>>
>> https://groups.google.com/d/msgid/prometheus-users/e5614621-f57a-466e-befd-269bf77d69c8o%40googlegroups.com.
>>
>>
>> > >>>
>> > >>> > >>>>
>> > >>> > >>>>
>> > >>> > >>>>
>> > >>> > >>>> --
>> > >>> > >>>> Julien Pivotto
>> > >>> > >>>> @roidelapluie
>> > >>> > >>>>
>> > >>> > >>>
>> > >>> >
>> > >>> > --
>> > >>> > You received this message because you are subscribed to the
>> Google
>> > >>> Groups "Prometheus Users" group.
>> > >>> > To unsubscribe from this group and stop receiving emails from it,
>> send
>> > >>> an email to [email protected].
>> > >>> > To view this discussion on the web visit
>> > >>>
>> https://groups.google.com/d/msgid/prometheus-users/e058c64f-3db4-45c2-9550-c8db557d2a2cn%40googlegroups.com.
>>
>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Julien Pivotto
>> > >>> @roidelapluie
>> > >>>
>> > >> --
>> > >> You received this message because you are subscribed to the Google
>> Groups
>> > >> "Prometheus Users" group.
>> > >> To unsubscribe from this group and stop receiving emails from it,
>> send an
>> > >> email to [email protected].
>> > >>
>> > > To view this discussion on the web visit
>> > >>
>> https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com
>>
>> > >> <
>> https://groups.google.com/d/msgid/prometheus-users/50d9a66e-5319-41a6-83ff-1836d86272d3n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>
>>
>> > >> .
>> > >>
>> > >
>> > >
>> > > --
>> > > Alexandru Duzsardi,
>> > > *DevOps Engineer*
>> > > *Skype:* alexinno83
>> > > *GPG/PGP Key*: https://keybase.io/aduzsardi/pgp_keys.asc
>> > > *GitLab:* https://gitlab.com/aduzsardi
>> > > *GitHub:* https://github.com/aduzsardi
>> > > *LinkedIn:* https://www.linkedin.com/in/aduzsardi
>> > > *E-mail:* [email protected]
>> > >
>> > > InFinIT Partners,
>> > > *Address:* Str. Macinului Nr. 17, Cluj-Napoca, Romania
>> > > *Web:* www.infinitpartners.com
>> > >
>> > >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to [email protected].
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/f1962c97-b545-47e8-9877-3482bdfaadean%40googlegroups.com.
>>
>>
>>
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/135b6338-030d-4f76-b75d-3038a7c98ec7n%40googlegroups.com.
