Hello list member, after upgrading from Octopus to Quincy yesterday, now we have a problem starting OSDs on the newest Rocky 8.6 4.18.0-372.26.1.el8_6.x86_64.
This is a non-cephadm Cluster. All nodes running Rocky with Kernel 4.18.0-372.19.1.el8_6.x86_64 except this one host (ceph1n012) i restarted shortly After rebooting the host to the new Kernel there were no OSD tmpfs mounts visible under /var/lib/ceph/osd/ceph-xy -- Unit [email protected] has begun starting up. Sep 29 15:33:21 ceph1n012 systemd[1]: Started Ceph object storage daemon osd.26. -- Subject: Unit [email protected] has finished start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Unit [email protected] has finished starting up. -- The start-up result is done. Sep 29 15:33:21 ceph1n012 ceph-osd[2258]: 2022-09-29T15:33:21.754+0200 7f49126663c0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-26/keyring: (2) No such file or directory Sep 29 15:33:21 ceph1n012 ceph-osd[2258]: 2022-09-29T15:33:21.754+0200 7f49126663c0 -1 AuthRegistry(0x56100fdb4140) no keyring found at /var/lib/ceph/osd/ceph-26/keyring, disabling cephx Sep 29 15:33:21 ceph1n012 ceph-osd[2258]: 2022-09-29T15:33:21.755+0200 7f49126663c0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-26/keyring: (2) No such file or directory Sep 29 15:33:21 ceph1n012 ceph-osd[2258]: 2022-09-29T15:33:21.755+0200 7f49126663c0 -1 AuthRegistry(0x7fff3eb59610) no keyring found at /var/lib/ceph/osd/ceph-26/keyring, disabling cephx Sep 29 15:33:21 ceph1n012 ceph-osd[2258]: failed to fetch mon config (--no-mon-config to skip) Sep 29 15:33:21 ceph1n012 systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE Sep 29 15:33:21 ceph1n012 systemd[1]: [email protected]: Failed with result 'exit-code'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- The unit [email protected] has entered the 'failed' state with result 'exit-code'. We also get these warning from systemd: [root@ceph1n012 ~]# systemctl status ceph-osd@24 \u25cf [email protected] - Ceph object storage daemon osd.24 Loaded: loaded (/usr/lib/systemd/system/[email protected]; disabled; vendor preset: disabled) Active: inactive (dead) *Sep 29 08:43:39 ceph1n012 systemd[1]: /usr/lib/systemd/system/[email protected]:23: Unknown lvalue 'ProtectHostname' in section 'Service'Sep 29 08:43:39 ceph1n012 systemd[1]: /usr/lib/systemd/system/[email protected]:24: Unknown lvalue 'ProtectKernelLogs' in section 'Service* It seems that the OSD process directly stops and there are no logfiles written so far.... Does someone have an eplanation for this bevaviour? What is the cause missing tmpfs mounts an who mounts these? systemd, the OSD itself or..?? Many thanks for any hint helping to get missing 7 OSDs up ASAP. Christoph cluster: id: a9dd5dd5-f87c-4a42-a251-fa6d21934914 health: HEALTH_WARN noout,norecover flag(s) set 7 osds down 2 hosts (7 osds) down Degraded data redundancy: 12808072/127922532 objects degraded (10.012%), 650 pgs degraded, 762 pgs undersized services: mon: 5 daemons, quorum ceph1n011,ceph1n021,ceph1n020,ceph1n019,ceph1n012 (age 73m) mgr: ceph1n019(active, since 26h), standbys: ceph1n020, ceph1n021 mds: 8/8 daemons up, 3 standby osd: 79 osds: 72 up (since 78m), 79 in (since 7w) flags noout,norecover { "mon": { "ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)": 5 }, "mgr": { "ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)": 3 }, "osd": { "ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)": 72 }, "mds": { "ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)": 3 }, "overall": { "ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)": 83 Christoph Ackermann | System Engineer INFOSERVE GmbH | Am Felsbrunnen 15 | D-66119 Saarbrücken Fon +49 (0)681 88008-59 | Fax +49 (0)681 88008-33 | [email protected] | www.infoserve.de INFOSERVE Datenschutzhinweise: www.infoserve.de/datenschutz Handelsregister: Amtsgericht Saarbrücken, HRB 11001 | Erfüllungsort: Saarbrücken Geschäftsführer: Dr. Stefan Leinenbach | Ust-IdNr.: DE168970599 <https://facebook.com/infoserve.de> <https://www.xing.com/companies/infoservegmbh> <https://www.youtube.com/channel/UCUj8C3TGGhQZPVvxu4woXmQ> <https://www.linkedin.com/company-beta/10095540> _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
