Huh, that's kind of interesting... So yes, we do have a
container_image override in our production cluster because of a
private container registry, but not in that test cluster.
I wiped that single host test cluster completely (cephadm rm-cluster)
and bootstrapped a fresh one. That's where I previously got that
grafana version from. Now I did that again, only this time I also
pruned all stored images on that host before bootstrapping again. And
now I get a different grafana version:
root@ceph0-2:~# ceph orch ps --daemon-type grafana --format json | jq
-r '.[].version'
12.3.1
That is unexpected (to me) but good to know not to rely on that. I'll
update our grafana image as well, let's see if those issues are
resolved.
Thanks!
Eugen
Zitat von Anthony D'Atri <[email protected]>:
[ I've left off dev@ from this reply, as it doesn't seem necessarily
like a dev issue ]
Is it possible that your cluster is overriding the default container
images, assuming that you're running containerized with cephadm?
root@squid1:~# ceph config dump | grep container_image
container_image XX
mgr/cephadm/container_image_alertmanager YY
mgr/cephadm/container_image_base ZZ
mgr/cephadm/container_image_elasticsearch QQ
mgr/cephadm/container_image_grafana ..
mgr/cephadm/container_image_haproxy ..
mgr/cephadm/container_image_jaeger_agent ..
mgr/cephadm/container_image_jaeger_collector ..
mgr/cephadm/container_image_jaeger_query ..
mgr/cephadm/container_image_keepalived ..
mgr/cephadm/container_image_loki ..
mgr/cephadm/container_image_node_exporter ..
mgr/cephadm/container_image_nvmeof ..
mgr/cephadm/container_image_prometheus ..
mgr/cephadm/container_image_promtail ..
mgr/cephadm/container_image_samba ..
mgr/cephadm/container_image_snmp_gateway ..
On Jul 1, 2026, at 6:53 AM, Sake via ceph-users <[email protected]> wrote:
Probably something not going correctly, because I just upgraded to
19.2.4 and Grafana is using the grafana image (in the past this was
ceph-grafana) version 12.3.1.
On 1 July 2026 11:13:02 CEST, Eugen Block via ceph-users
<[email protected]> wrote:
Hi,
I just upgraded our production cluster to 19.2.4 (from 19.2.3),
there were no issues reported by our users, so all good. But I
noticed that many graphs now show different warnings or strange
labels:
- "This panel requires Angular (deprecated)." on multiple graphs
- "PromQL info: metric might not be a counter, name does not end
in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on
multiple graphs for different metrics
- "OSD Capacity Utilization" just shows: "RangeError: invalid array length"
- The "Pool Overview" graph has column names like "Value #F",
"Value #G" and so on. And the values within don't make sense either.
I have uploaded screenshots:
https://www.pasteboard.co/A34Zcy9cqHe5.png
https://www.pasteboard.co/F_vkpeBjGGWH.png
https://www.pasteboard.co/mkCtlBN3zS6v.png
https://www.pasteboard.co/T_ByD_gCpslA.png
I compared the prometheus and grafana versions we have currently
running with a fresh 19.2.4 bootstrapped cluster and the versions
are identical:
ceph orch ps --daemon-type grafana --format json | jq -r '.[].version'
10.4.0
ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version'
2.51.0
Is there anything I'm missing?
Thanks!
Eugen
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
