Hello Adam,
Thanks for the prompt response.
We have below image in private-registry for node-exporter.
192.168.1.10:5000/prometheus/node-exporter v1.5.0 0da6a335fe13
19 months ago 22.5MB
But upon ceph upgrade, we are getting the mentioned image (
quay.io/prometheus/node-exporter:v1.5.0) by executing following command.
root@node-01:~# ceph config get mgr mgr/cephadm/container_image_node_exporter
quay.io/prometheus/node-exporter:v1.5.0
I am a bit reluctant which image do i need to work upon, the image that is
there on private-registry (192.168.1.10:5000/prometheus/node-exporter ) or the
image that we are getting after executing "ceph config" command i.e (
quay.io/prometheus/node-exporter:v1.5.0 ).
For every ( monitoring stack components ) images do we required Internet no
matter where the image is present even if it is there on private-registry.
Right ?
We executed following command but it is in error state.
root@node-01:~# ceph config get mgr mgr/cephadm/container_image_node_exporter
192.168.1.10:5000/prometheus/node-exporter:v1.5.0
Sharing some important outputs:
root@node-01:~# ceph health detail
HEALTH_WARN 1 failed cephadm daemon(s)
[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
daemon node-exporter.sky-blue on sky-blue is in error state
Journalctl logs:
Jul 16 12:01:18 sky-blue systemd[1]:
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774@node-exporter. sky-blue.service:
Failed with result 'exit-code'.
Jul 16 12:01:18 sky-blue systemd[1]: Failed to start Ceph node-exporter.
sky-blue for 3d444cb6-435b-11ef-bdc9-f30dc5fa2774.
Jul 16 12:01:28 sky-blue systemd[1]:
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774@node-exporter. sky-blue.service:
Scheduled restart job, restart counter is at 4.
Jul 16 12:01:28 sky-blue systemd[1]: Stopped Ceph node-exporter. sky-blue for
3d444cb6-435b-11ef-bdc9-f30dc5fa2774.
Jul 16 12:01:28 sky-blue systemd[1]: Starting Ceph node-exporter. sky-blue for
3d444cb6-435b-11ef-bdc9-f30dc5fa2774...
Jul 16 12:01:29 sky-blue bash[115638]: Trying to pull
quay.io/prometheus/node-exporter:v1.5.0...
Jul 16 12:02:29 sky-blue bash[115638]: Error: initializing source
docker://quay.io/prometheus/node-exporter:v1.5.0: pinging container registry
quay.io: Get "https://quay.io/v2/": dial tcp 3.214.40.167:443: i/o timeout
Jul 16 12:02:29 sky-blue systemd[1]:
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774@node-exporter. sky-blue.service:
Control process exited, code=exited, status=125/n/a
Jul 16 12:02:29 sky-blue systemd[1]:
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774@node-exporter. sky-blue.service:
Failed with result 'exit-code'.
Jul 16 12:02:29 sky-blue systemd[1]: Failed to start Ceph node-exporter.
sky-blue for 3d444cb6-435b-11ef-bdc9-f30dc5fa2774.
Jul 16 12:02:39 sky-bluey systemd[1]:
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774@node-exporter. sky-blue.service:
Scheduled restart job, restart counter is at 5.
Jul 16 12:02:39 sky-blue systemd[1]: Stopped Ceph node-exporter. sky-blue for
3d444cb6-435b-11ef-bdc9-f30dc5fa2774.
Jul 16 12:02:39 sky-blue systemd[1]:
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774@node-exporter. sky-blue.service:
Start request repeated too quickly.
Jul 16 12:02:39 sky-blue systemd[1]:
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774@node-exporter. sky-blue.service:
Failed with result 'exit-code'.
Jul 16 12:02:39 sky-bluey systemd[1]: Failed to start Ceph node-exporter.
sky-blue for 3d444cb6-435b-11ef-bdc9-f30dc5fa2774.
It is attempting to start the following image:
quay.io/prometheus/node-exporter:v1.5.0
root@sky-blue:/var/lib/ceph/3d444cb6-435b-11ef-bdc9-f30dc5fa2774/node-exporter.sky-blue#
cat unit.run
set -e
# node-exporter.sky-blue
! /usr/bin/podman rm -f
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774-node-exporter.sky-blue 2> /dev/null
! /usr/bin/podman rm -f
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774-node-exporter-sky-blue 2> /dev/null
! /usr/bin/podman rm -f --storage
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774-node-exporter-sky-blue 2> /dev/null
! /usr/bin/podman rm -f --storage
ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774-node-exporter.sky-blue 2> /dev/null
/usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --init
--name ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2774-node-exporter-sky-blue --user
65534 --security-opt label=disable -d --log-driver journald --conmon-pidfile
/run/ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2...@node-exporter.sky-blue.service-pid
--cidfile
/run/ceph-3d444cb6-435b-11ef-bdc9-f30dc5fa2...@node-exporter.sky-blue.service-cid
--cgroups=split -e CONTAINER_IMAGE=quay.io/prometheus/node-exporter:v1.5.0 -e
NODE_NAME=ocean-bay -e CEPH_USE_RANDOM_NONCE=1 -v
/var/lib/ceph/3d444cb6-435b-11ef-bdc9-f30dc5fa2774/node-exporter.sky-blue/etc/node-exporter:/etc/node-exporter:Z
-v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /:/rootfs:ro -v
/etc/hosts:/etc/hosts:ro quay.io/prometheus/node-exporter:v1.5.0
--no-collector.timex --web.listen-address=:9100 --path.procfs=/host/proc
--path.sysfs=/host/sys --path.rootfs=/rootfs
Please guide how to resolve this.
Regards,
Mohammad Saif
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
