Hi,
not sure if it was related to journal replay, but have you checked for
memory issues? What's the mds memory target? Any traces of an oom
killer?
Next I would do is inspect the journals for both purge_queue and md_log:
cephfs-journal-tool journal inspect --rank=<cephfs> --journal=md_log
cephfs-journal-tool journal inspect --rank=<cephfs> --journal=purge_queue
The --rank and --journal parameters might be in the wrong place here,
I'm writing this without immediate access to a cephfs-journal-tool.
In case the journals are okay, create a backup as described in the
docs [0]. Then you might have to go through the disaster recovery
steps (for this cephfs only).
[0] https://docs.ceph.com/en/latest/cephfs/disaster-recovery/
Zitat von Kasper Rasmussen <kasper_steenga...@hotmail.com>:
> Ceph Version: 18.2.7
>
> I've just migrated to cephadm, and upgrade from pacific to reef
> 18.2.7 last week.
> All successful except some minor issues with BlueFS Spillover
>
>
> Today the MDS of a specific fs refuse to start, and the ceph orch ps
> shows the daemons with status "error".
> I have three other cephfs that still works(though I haven't tested
> if they can fail over.)
>
> I've restartet the MDSs - No luck (the selected MDS just start/crash
> in a loop until it gives up)
> I've deployed 2 new MDSs - No luck same issue
>
> In all scenarios I see in ceph fs status, that a MDS is chosen. FS
> status goes to "replay" or "replay(laggy)"
> On the host with the MDS I see the MDS container just crashes after
> way less than 5 mins.. And status reported by ceph orch ps is error.
>
> (btw - mds_beacon_grace has been set to 360)
>
> I've managed to get a good 500 lines of log out with info like this:
>
> << ----------------- LOG EXAMPLE START ----------------- >>
> -7> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient:
> _check_auth_tickets
> -6> 2025-05-19T16:05:02.840+0000 7f6739bb8640 10 monclient:
> _check_auth_rotating have uptodate secrets (they expire after
> 2025-05-19T16:04:32.845551+0000)
> -5> 2025-05-19T16:05:02.860+0000 7f673e3c1640 10 monclient:
> get_auth_request con 0x5616e9616c00 auth_method 0
> -4> 2025-05-19T16:05:02.916+0000 7f673dbc0640 10 monclient:
> get_auth_request con 0x5616e7422800 auth_method 0
> -3> 2025-05-19T16:05:02.968+0000 7f673d3bf640 10 monclient:
> get_auth_request con 0x5616f5eac800 auth_method 0
> -2> 2025-05-19T16:05:02.972+0000 7f6736bb2640 2 mds.0.cache
> Memory usage: total 574800, rss 343772, heap 207124, baseline
> 182548, 0 / 7535 inodes have caps, 0 caps, 0 caps per inode
> -1> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: In function 'void interval_set<T, C>::erase(T, T, std::function<bool(T, T)>) [with T = inodeno_t; C = std::map]' thread 7f67333ab640
time
> 2025-05-19T16:05:03.680495+0000
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.7/rpm/el9/BUILD/ceph-18.2.7/src/include/interval_set.h: 568: FAILED ceph_assert(p->first
<=
> start)
>
> ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad)
reef (stable)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x11e) [0x7f67406e6d2c]
> 2: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb]
> 3: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe]
> 4: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745]
> 5: (EMetaBlob::replay(MDSRank*, LogSegment*, int,
> MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c]
> 6: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd]
> 7: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e]
> 8: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1]
> 9: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a]
> 10: clone()
>
> 0> 2025-05-19T16:05:03.676+0000 7f67333ab640 -1 *** Caught
> signal (Aborted) **
> in thread 7f67333ab640 thread_name:mds-log-replay
>
> ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad)
reef (stable)
> 1: /lib64/libc.so.6(+0x3ebf0) [0x7f674004bbf0]
> 2: /lib64/libc.so.6(+0x8bf5c) [0x7f6740098f5c]
> 3: raise()
> 4: abort()
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x178) [0x7f67406e6d86]
> 6: /usr/lib64/ceph/libceph-common.so.2(+0x16beeb) [0x7f67406e6eeb]
> 7: /usr/bin/ceph-mds(+0x1f16fe) [0x5616e04d46fe]
> 8: /usr/bin/ceph-mds(+0x1f1745) [0x5616e04d4745]
> 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int,
> MDPeerUpdate*)+0x4bdc) [0x5616e0709a4c]
> 10: (EUpdate::replay(MDSRank*)+0x5d) [0x5616e0711afd]
> 11: (MDLog::_replay_thread()+0x75e) [0x5616e06bc02e]
> 12: /usr/bin/ceph-mds(+0x1404b1) [0x5616e04234b1]
> 13: /lib64/libc.so.6(+0x8a21a) [0x7f674009721a]
> 14: clone()
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
> << ----------------- LOG EXAMPLE END ----------------- >>
>
>
> But to be honest, out of all those lines, I don't know what to
> provide (all +500 might be a bit to much)
>
>
> I really need this FS back online, so help will be very much appreciated
>
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
