Use journalctl -xe (maybe with -S/-U if you want to filter) to find the time period in which a restart attempt has happened, and see what's logged at that period. If that's not helpful, then what you may want to do is disable that service (systemctl disable blah) then get the ExecStart out of it, then try running it by hand and seeing what happens (the symlinked systemd unit will will refer to a unit.run file in /var/lib/ceph that will have the actual podman cmd). If the pod dies, you should still see it in podman ps -a and you can podman logs on it to get the details. Then you can correct the issue then re-enable the service and restart it properly to do the housekeeping. Follow these directions at your own risk; make sure you understand the ramifications of whatever you might be doing!
David On Thu, Mar 18, 2021 at 3:29 PM Philip Brown <[email protected]> wrote: > > I've been banging on my ceph octopus test cluster for a few days now. > 8 nodes. each node has 2 SSDs and 8 HDDs. > They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as > a db partition. > > service_type: osd > service_id: osd_spec_default > placement: > host_pattern: '*' > data_devices: > rotational: 1 > db_devices: > rotational: 0 > > > things were going pretty good, until... yesterday.. i noticed TWO of the OSDs > were "down". > > I went to check the logs, with > journalctl -u [email protected] > > all it showed were a bunch of generic debug info, and the fact that it > stopped. > and various automatic attempts to restart. > but no indication of what was wrong, and why the restarts KEEP failing. > > > sample output: > > > systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00. > systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00... > bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate > bash[9340]: WARNING: The same type, major and minor should not be used for > multiple devices. > bash[9340]: WARNING: The same type, major and minor should not be used for > multiple devices. > podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 > container create > podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container > init > ..... > bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33 > podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 > container died > bash[1611473]: ceph-xx-xx-xx-xx-osd.33 > bash[1611473]: WARNING: The same type, major and minor should not be used for > multiple devices. > (repeat, repeat...) > podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 > container create > > .... > systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx > systemd[1]: [email protected]: main process exited, > code=exited, status=1/FAILURE > bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate > > and eventually it just gives up. > > smartctl -a doesnt show any errors on the HDD > > > dmesg doesnt show anything. > > So... what do I do? > > > > > > -- > Philip Brown| Sr. Linux System Administrator | Medata, Inc. > 5 Peters Canyon Rd Suite 250 > Irvine CA 92606 > Office 714.918.1310| Fax 714.918.1325 > [email protected]| www.medata.com > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
