Re: [ceph-users] cephfs: how to repair damaged mds rank?

[Daniel Baumann]

Hi John,

On 10/09/2017 10:47 AM, John Spray wrote:
> When a rank is "damaged", that means the MDS rank is blocked from
> starting because Ceph thinks the on-disk metadata is damaged -- no
> amount of restarting things will help.

thanks.

> The place to start with the investigation is to find the source of the
> damage.  Look in your monitor log for "marking rank 6 damaged"

I found this in the mon log:

  2017-10-09 03:24:28.207424 7f3290710700  0 log_channel(cluster) log
 [DBG] : mds.6 147.87.226.187:6800/1120166215 down:damaged

so at the time it was marked damaged, rank 6 was running on mds7.

> and then look in your MDS logs at that timestamp (find the MDS that held
> rank 6 at the time).

looking at mds7 log for that timespan, I think I understand that:

  * at "early" 03:24, mds7 was serving rank 5 and crashed, restarted
automatically twice, and then picked up rank 6 at 03:24:21.

  * at 03:24:21, mds7 got rank 6 and got into 'standby'-mode(?):

2017-10-09 03:24:21.598446 7f70ca01c240  0 set uid:gid to 64045:64045
(ceph:ceph)
2017-10-09 03:24:21.598469 7f70ca01c240  0 ceph version 12.2.1
(3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process
(unknown), pid 1337
2017-10-09 03:24:21.601958 7f70ca01c240  0 pidfile_write: ignore empty
--pid-file
2017-10-09 03:24:26.108545 7f70c2580700  1 mds.mds7 handle_mds_map standby
2017-10-09 03:24:26.115469 7f70c2580700  1 mds.6.95474 handle_mds_map i
am now mds.6.95474
2017-10-09 03:24:26.115479 7f70c2580700  1 mds.6.95474 handle_mds_map
state change up:boot --> up:replay
2017-10-09 03:24:26.115493 7f70c2580700  1 mds.6.95474 replay_start
2017-10-09 03:24:26.115502 7f70c2580700  1 mds.6.95474  recovery set is
0,1,2,3,4,5,7,8
2017-10-09 03:24:26.115511 7f70c2580700  1 mds.6.95474  waiting for
osdmap 18284 (which blacklists prior instance)
2017-10-09 03:24:26.536629 7f70bc574700  0 mds.6.cache creating system
inode with ino:0x106
2017-10-09 03:24:26.537009 7f70bc574700  0 mds.6.cache creating system
inode with ino:0x1
2017-10-09 03:24:27.233759 7f70bd576700 -1 mds.6.journaler.pq(ro)
_decode error from assimilate_prefetch
2017-10-09 03:24:27.233780 7f70bd576700 -1 mds.6.purge_queue _recover:
Error -22 recovering write_pos
2017-10-09 03:24:27.238820 7f70bd576700  1 mds.mds7 respawn
2017-10-09 03:24:27.238828 7f70bd576700  1 mds.mds7  e: '/usr/bin/ceph-mds'
2017-10-09 03:24:27.238831 7f70bd576700  1 mds.mds7  0: '/usr/bin/ceph-mds'
2017-10-09 03:24:27.238833 7f70bd576700  1 mds.mds7  1: '-f'
2017-10-09 03:24:27.238835 7f70bd576700  1 mds.mds7  2: '--cluster'
2017-10-09 03:24:27.238836 7f70bd576700  1 mds.mds7  3: 'ceph'
2017-10-09 03:24:27.238838 7f70bd576700  1 mds.mds7  4: '--id'
2017-10-09 03:24:27.238839 7f70bd576700  1 mds.mds7  5: 'mds7'
2017-10-09 03:24:27.239567 7f70bd576700  1 mds.mds7  6: '--setuser'
2017-10-09 03:24:27.239579 7f70bd576700  1 mds.mds7  7: 'ceph'
2017-10-09 03:24:27.239580 7f70bd576700  1 mds.mds7  8: '--setgroup'
2017-10-09 03:24:27.239581 7f70bd576700  1 mds.mds7  9: 'ceph'
2017-10-09 03:24:27.239612 7f70bd576700  1 mds.mds7 respawning with exe
/usr/bin/ceph-mds
2017-10-09 03:24:27.239614 7f70bd576700  1 mds.mds7  exe_path /proc/self/exe
2017-10-09 03:24:27.268448 7f9c7eafa240  0 ceph version 12.2.1
(3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process
(unknown), pid 1337
2017-10-09 03:24:27.271987 7f9c7eafa240  0 pidfile_write: ignore empty
--pid-file
2017-10-09 03:24:31.325891 7f9c7789c700  1 mds.mds7 handle_mds_map standby
2017-10-09 03:24:31.332376 7f9c7789c700  1 mds.1.0 handle_mds_map i am
now mds.28178286.0 replaying mds.1.0
2017-10-09 03:24:31.332388 7f9c7789c700  1 mds.1.0 handle_mds_map state
change up:boot --> up:standby-replay
2017-10-09 03:24:31.332401 7f9c7789c700  1 mds.1.0 replay_start
2017-10-09 03:24:31.332410 7f9c7789c700  1 mds.1.0  recovery set is
0,2,3,4,5,6,7,8
2017-10-09 03:24:31.332425 7f9c7789c700  1 mds.1.0  waiting for osdmap
18285 (which blacklists prior instance)
2017-10-09 03:24:31.351850 7f9c7108f700  0 mds.1.cache creating system
inode with ino:0x101
2017-10-09 03:24:31.352204 7f9c7108f700  0 mds.1.cache creating system
inode with ino:0x1
2017-10-09 03:24:32.144505 7f9c7008d700  0 mds.1.cache creating system
inode with ino:0x100
2017-10-09 03:24:32.144671 7f9c7008d700  1 mds.1.0 replay_done (as standby)
2017-10-09 03:24:33.150117 7f9c71890700  1 mds.1.0 replay_done (as standby)

for about two hours, then, the last line repeats unchanged for every
following second.

where can I go with this? anything I can do further?

also, just in case: it seems that at the time of the crash a large (= a
lot, lot of small files) 'rm -rf' was running (all clients use kernel
4.13.4 to mount the cephfs, not fuse).

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs: how to repair damaged mds rank?

[John Spray]

On Mon, Oct 9, 2017 at 8:17 AM, Daniel Baumann  wrote:
> Hi all,
>
> we have a Ceph Cluster (12.2.1) with 9 MDS ranks in multi-mds mode.
>
> "out of the blue", rank 6 is marked as damaged (and all other MDS are in
> state up:resolve) and I can't bring the FS up again.
>
> 'ceph -s' says:
> [...]
> 1 filesystem is degraded
> 1 mds daemon damaged
>
> mds: cephfs-8/9/9 up
> {0=mds1=up:resolve,1=mds2=up:resolve,2=mds3=up:resolve,3=mds4=up:resolve,4=mds5=up:resolve,5=mds6=up:resolve,7=mds7=
> up:resolve,8=mds8=up:resolve}, 1 up:standby, 1 damaged
> [...]
>
> 'ceph fs get cephfs' says:
> [...]
> max_mds 9
> in  0,1,2,3,4,5,6,7,8
> up
> {0=28309098,1=28309128,2=28309149,3=28309188,4=28309209,5=28317918,7=28311732,8=28312272}
> failed
> damaged 6
> stopped
> [...]
> 28309098:   147.87.226.60:6800/2627352929 'mds1' mds.0.95936
> up:resolve seq 3
> 28309128:   147.87.226.61:6800/416822271 'mds2' mds.1.95939
> up:resolve seq 3
> 28309149:   147.87.226.62:6800/1969015920 'mds3' mds.2.95942
> up:resolve seq 3
> 28309188:   147.87.226.184:6800/4074580566 'mds4' mds.3.95945
> up:resolve seq 3
> 28309209:   147.87.226.185:6800/805082194 'mds5' mds.4.95948
> up:resolve seq 3
> 28317918:   147.87.226.186:6800/1913199036 'mds6' mds.5.95984
> up:resolve seq 3
> 28311732:   147.87.226.187:6800/4117561729 'mds7' mds.7.95957
> up:resolve seq 3
> 28312272:   147.87.226.188:6800/2936268159 'mds8' mds.8.95960
> up:resolve seq 3
>
>
> I think I've tried almost anything already without success :(, including:
>
>   * stopping all MDS, and bringing them up one after one
> (works nice for the first ones up to rank 5, then the next one
>  just grabs rank 7 and no MDS after that wants to take rank 6)
>
>   * stopped all MDS, flushed MDS journal, manually marked rank 6 as
> repaired, started all MDS again.
>
>   * tried to switch back to only one MDS (stopping all MDS, setting
> max_mds=1, disallowing multi-mds, disallowing dirfrag, removing
> "mds_bal_frag=true" from ceph.conf, then starting the first mds),
> didn't work.. the one single MDS stayed in up:resolve forever.
>
>   * during all of the above, all CephFS clients have been unmounted,
> so there's no access/stale access to the FS
>
>   * did find a few things in the mailinglist archive, but seems there's
> nothing conclusive on how to get it back online ("formating" the
> FS is not possible). I didn't dare trying 'ceph mds rmfailed 6'
> in fear of dataloss.
>
>
> How can I get it back online?

When a rank is "damaged", that means the MDS rank is blocked from
starting because Ceph thinks the on-disk metadata is damaged -- no
amount of restarting things will help.

The place to start with the investigation is to find the source of the
damage.  Look in your monitor log for "marking rank 6 damaged", and
then look in your MDS logs at that timestamp (find the MDS that held
rank 6 at the time).

John

> The relevant portion from the ceph-mds log (when starting mds9 which
> should then take up rank 6; I'm happy to provide any logs):
>
> ---snip---
> 2017-10-09 08:55:56.418237 7f1ec6ef3240  0 ceph version 12.2.1
> (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process
> (unknown), pid 421
> 2017-10-09 08:55:56.421672 7f1ec6ef3240  0 pidfile_write: ignore empty
> --pid-file
> 2017-10-09 08:56:00.990530 7f1ebf457700  1 mds.mds9 handle_mds_map standby
> 2017-10-09 08:56:00.997044 7f1ebf457700  1 mds.6.95988 handle_mds_map i
> am now mds.6.95988
> 2017-10-09 08:56:00.997053 7f1ebf457700  1 mds.6.95988 handle_mds_map
> state change up:boot --> up:replay
> 2017-10-09 08:56:00.997068 7f1ebf457700  1 mds.6.95988 replay_start
> 2017-10-09 08:56:00.997076 7f1ebf457700  1 mds.6.95988  recovery set is
> 0,1,2,3,4,5,7,8
> 2017-10-09 08:56:01.003203 7f1eb8c4a700  0 mds.6.cache creating system
> inode with ino:0x106
> 2017-10-09 08:56:01.003592 7f1eb8c4a700  0 mds.6.cache creating system
> inode with ino:0x1
> 2017-10-09 08:56:01.016403 7f1eba44d700 -1 mds.6.journaler.pq(ro)
> _decode error from assimilate_prefetch
> 2017-10-09 08:56:01.016425 7f1eba44d700 -1 mds.6.purge_queue _recover:
> Error -22 recovering write_pos
> 2017-10-09 08:56:01.019746 7f1eba44d700  1 mds.mds9 respawn
> 2017-10-09 08:56:01.019762 7f1eba44d700  1 mds.mds9  e: '/usr/bin/ceph-mds'
> 2017-10-09 08:56:01.019765 7f1eba44d700  1 mds.mds9  0: '/usr/bin/ceph-mds'
> 2017-10-09 08:56:01.019767 7f1eba44d700  1 mds.mds9  1: '-f'
> 2017-10-09 08:56:01.019769 7f1eba44d700  1 mds.mds9  2: '--cluster'
> 2017-10-09 08:56:01.019771 7f1eba44d700  1 mds.mds9  3: 'ceph'
> 2017-10-09 08:56:01.019772 7f1eba44d700  1 mds.mds9  4: '--id'
> 2017-10-09 08:56:01.019773 7f1eba44d700  1 mds.mds9  5: 'mds9'
> 2017-10-09 08:56:01.019774 7f1eba44d700  1 mds.mds9  6: '--setuser'
> 2017-10-09 08:56:01.019775 7f1eba44d700  1 mds.mds9  7: 'ceph'
> 2017-10-09 08:56:01.019776 7f1eba44d700  1 mds.mds9  8: '--setgroup'
> 

Re: [ceph-users] cephfs: how to repair damaged mds rank?

[Daniel Baumann]

On 10/09/2017 09:17 AM, Daniel Baumann wrote:
> The relevant portion from the ceph-mds log (when starting mds9 which
> should then take up rank 6; I'm happy to provide any logs):

i've turned up the logging (see attachment).. could it be that we hit
this bug here?

http://tracker.ceph.com/issues/17670

Regards,
Daniel
2017-10-09 10:07:14.677308 7f7972bd6700 10 mds.beacon.mds9 handle_mds_beacon up:standby seq 6 rtt 0.000642
2017-10-09 10:07:15.547453 7f7972bd6700  5 mds.mds9 handle_mds_map epoch 96022 from mon.0
2017-10-09 10:07:15.547526 7f7972bd6700 10 mds.mds9  my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in om
ap,7=mds uses inline data,8=file layout v2}
2017-10-09 10:07:15.547546 7f7972bd6700 10 mds.mds9  mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in om
ap,8=file layout v2}
2017-10-09 10:07:15.547555 7f7972bd6700 10 mds.mds9 map says I am 147.87.226.189:6800/6621615 mds.6.96022 state up:replay
2017-10-09 10:07:15.547825 7f7972bd6700  4 mds.6.purge_queue operator():  data pool 7 not found in OSDMap
2017-10-09 10:07:15.547882 7f7972bd6700 10 mds.mds9 handle_mds_map: initializing MDS rank 6
2017-10-09 10:07:15.548165 7f7972bd6700 10 mds.6.0 update_log_config log_to_monitors {default=true}
2017-10-09 10:07:15.548171 7f7972bd6700 10 mds.6.0 create_logger
2017-10-09 10:07:15.548410 7f7972bd6700  7 mds.6.server operator(): full = 0 epoch = 0
2017-10-09 10:07:15.548423 7f7972bd6700  4 mds.6.purge_queue operator():  data pool 7 not found in OSDMap
2017-10-09 10:07:15.548427 7f7972bd6700  4 mds.6.0 handle_osd_map epoch 0, 0 new blacklist entries
2017-10-09 10:07:15.548439 7f7972bd6700 10 mds.6.server apply_blacklist: killed 0
2017-10-09 10:07:15.548634 7f7972bd6700 10 mds.mds9 handle_mds_map: handling map as rank 6
2017-10-09 10:07:15.548647 7f7972bd6700  1 mds.6.96022 handle_mds_map i am now mds.6.96022
2017-10-09 10:07:15.548650 7f7972bd6700  1 mds.6.96022 handle_mds_map state change up:boot --> up:replay
2017-10-09 10:07:15.548668 7f7972bd6700 10 mds.beacon.mds9 set_want_state: up:standby -> up:replay
2017-10-09 10:07:15.548687 7f7972bd6700  1 mds.6.96022 replay_start
2017-10-09 10:07:15.548699 7f7972bd6700  7 mds.6.cache set_recovery_set 0,1,2,3,4,5,7,8
2017-10-09 10:07:15.548706 7f7972bd6700  1 mds.6.96022  recovery set is 0,1,2,3,4,5,7,8
2017-10-09 10:07:15.548720 7f7972bd6700  1 mds.6.96022  waiting for osdmap 18484 (which blacklists prior instance)
2017-10-09 10:07:15.548737 7f7972bd6700  4 mds.6.purge_queue operator():  data pool 7 not found in OSDMap
2017-10-09 10:07:15.549521 7f7972bd6700  7 mds.6.server operator(): full = 0 epoch = 18492
2017-10-09 10:07:15.549534 7f7972bd6700  4 mds.6.96022 handle_osd_map epoch 18492, 0 new blacklist entries
2017-10-09 10:07:15.549537 7f7972bd6700 10 mds.6.server apply_blacklist: killed 0
2017-10-09 10:07:15.549582 7f796cbca700 10 MDSIOContextBase::complete: 12C_IO_Wrapper
2017-10-09 10:07:15.549679 7f796cbca700 10 MDSInternalContextBase::complete: 15C_MDS_BootStart
2017-10-09 10:07:15.549685 7f796cbca700  2 mds.6.96022 boot_start 0: opening inotable
2017-10-09 10:07:15.549695 7f796cbca700 10 mds.6.inotable: load
2017-10-09 10:07:15.549880 7f796cbca700  2 mds.6.96022 boot_start 0: opening sessionmap
2017-10-09 10:07:15.549888 7f796cbca700 10 mds.6.sessionmap load
2017-10-09 10:07:15.549977 7f796cbca700  2 mds.6.96022 boot_start 0: opening mds log
2017-10-09 10:07:15.549984 7f796cbca700  5 mds.6.log open discovering log bounds
2017-10-09 10:07:15.550113 7f796c3c9700  4 mds.6.journalpointer Reading journal pointer '406.'
2017-10-09 10:07:15.550132 7f796bbc8700 10 mds.6.log _submit_thread start
2017-10-09 10:07:15.551165 7f796cbca700 10 MDSIOContextBase::complete: 12C_IO_MT_Load
2017-10-09 10:07:15.551178 7f796cbca700 10 mds.6.inotable: load_2 got 34 bytes
2017-10-09 10:07:15.551184 7f796cbca700 10 mds.6.inotable: load_2 loaded v1
2017-10-09 10:07:15.565382 7f796cbca700 10 MDSIOContextBase::complete: N12_GLOBAL__N_112C_IO_SM_LoadE
2017-10-09 10:07:15.565397 7f796cbca700 10 mds.6.sessionmap _load_finish loaded version 0
2017-10-09 10:07:15.565401 7f796cbca700 10 mds.6.sessionmap _load_finish: omap load complete
2017-10-09 10:07:15.565403 7f796cbca700 10 mds.6.sessionmap _load_finish: v 0, 0 sessions
2017-10-09 10:07:15.565408 7f796cbca700 10 mds.6.sessionmap dump
2017-10-09 10:07:15.583721 7f796c3c9700  1 mds.6.journaler.mdlog(ro) recover start
2017-10-09 10:07:15.583732 7f796c3c9700  1 mds.6.journaler.mdlog(ro) read_head
2017-10-09 10:07:15.583854 7f796c3c9700  4 mds.6.log Waiting for journal 0x206 to recover...
2017-10-09 10:07:15.796523 7f796cbca700  1 mds.6.journaler.mdlog(ro) _finish_read_head loghead(trim 25992101888, expire 25992101888, write 25992101888, 

[ceph-users] cephfs: how to repair damaged mds rank?

[Daniel Baumann]

Hi all,

we have a Ceph Cluster (12.2.1) with 9 MDS ranks in multi-mds mode.

"out of the blue", rank 6 is marked as damaged (and all other MDS are in
state up:resolve) and I can't bring the FS up again.

'ceph -s' says:
[...]
1 filesystem is degraded
1 mds daemon damaged

mds: cephfs-8/9/9 up
{0=mds1=up:resolve,1=mds2=up:resolve,2=mds3=up:resolve,3=mds4=up:resolve,4=mds5=up:resolve,5=mds6=up:resolve,7=mds7=
up:resolve,8=mds8=up:resolve}, 1 up:standby, 1 damaged
[...]

'ceph fs get cephfs' says:
[...]
max_mds 9
in  0,1,2,3,4,5,6,7,8
up
{0=28309098,1=28309128,2=28309149,3=28309188,4=28309209,5=28317918,7=28311732,8=28312272}
failed
damaged 6
stopped
[...]
28309098:   147.87.226.60:6800/2627352929 'mds1' mds.0.95936
up:resolve seq 3
28309128:   147.87.226.61:6800/416822271 'mds2' mds.1.95939
up:resolve seq 3
28309149:   147.87.226.62:6800/1969015920 'mds3' mds.2.95942
up:resolve seq 3
28309188:   147.87.226.184:6800/4074580566 'mds4' mds.3.95945
up:resolve seq 3
28309209:   147.87.226.185:6800/805082194 'mds5' mds.4.95948
up:resolve seq 3
28317918:   147.87.226.186:6800/1913199036 'mds6' mds.5.95984
up:resolve seq 3
28311732:   147.87.226.187:6800/4117561729 'mds7' mds.7.95957
up:resolve seq 3
28312272:   147.87.226.188:6800/2936268159 'mds8' mds.8.95960
up:resolve seq 3


I think I've tried almost anything already without success :(, including:

  * stopping all MDS, and bringing them up one after one
(works nice for the first ones up to rank 5, then the next one
 just grabs rank 7 and no MDS after that wants to take rank 6)

  * stopped all MDS, flushed MDS journal, manually marked rank 6 as
repaired, started all MDS again.

  * tried to switch back to only one MDS (stopping all MDS, setting
max_mds=1, disallowing multi-mds, disallowing dirfrag, removing
"mds_bal_frag=true" from ceph.conf, then starting the first mds),
didn't work.. the one single MDS stayed in up:resolve forever.

  * during all of the above, all CephFS clients have been unmounted,
so there's no access/stale access to the FS

  * did find a few things in the mailinglist archive, but seems there's
nothing conclusive on how to get it back online ("formating" the
FS is not possible). I didn't dare trying 'ceph mds rmfailed 6'
in fear of dataloss.


How can I get it back online?

The relevant portion from the ceph-mds log (when starting mds9 which
should then take up rank 6; I'm happy to provide any logs):

---snip---
2017-10-09 08:55:56.418237 7f1ec6ef3240  0 ceph version 12.2.1
(3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process
(unknown), pid 421
2017-10-09 08:55:56.421672 7f1ec6ef3240  0 pidfile_write: ignore empty
--pid-file
2017-10-09 08:56:00.990530 7f1ebf457700  1 mds.mds9 handle_mds_map standby
2017-10-09 08:56:00.997044 7f1ebf457700  1 mds.6.95988 handle_mds_map i
am now mds.6.95988
2017-10-09 08:56:00.997053 7f1ebf457700  1 mds.6.95988 handle_mds_map
state change up:boot --> up:replay
2017-10-09 08:56:00.997068 7f1ebf457700  1 mds.6.95988 replay_start
2017-10-09 08:56:00.997076 7f1ebf457700  1 mds.6.95988  recovery set is
0,1,2,3,4,5,7,8
2017-10-09 08:56:01.003203 7f1eb8c4a700  0 mds.6.cache creating system
inode with ino:0x106
2017-10-09 08:56:01.003592 7f1eb8c4a700  0 mds.6.cache creating system
inode with ino:0x1
2017-10-09 08:56:01.016403 7f1eba44d700 -1 mds.6.journaler.pq(ro)
_decode error from assimilate_prefetch
2017-10-09 08:56:01.016425 7f1eba44d700 -1 mds.6.purge_queue _recover:
Error -22 recovering write_pos
2017-10-09 08:56:01.019746 7f1eba44d700  1 mds.mds9 respawn
2017-10-09 08:56:01.019762 7f1eba44d700  1 mds.mds9  e: '/usr/bin/ceph-mds'
2017-10-09 08:56:01.019765 7f1eba44d700  1 mds.mds9  0: '/usr/bin/ceph-mds'
2017-10-09 08:56:01.019767 7f1eba44d700  1 mds.mds9  1: '-f'
2017-10-09 08:56:01.019769 7f1eba44d700  1 mds.mds9  2: '--cluster'
2017-10-09 08:56:01.019771 7f1eba44d700  1 mds.mds9  3: 'ceph'
2017-10-09 08:56:01.019772 7f1eba44d700  1 mds.mds9  4: '--id'
2017-10-09 08:56:01.019773 7f1eba44d700  1 mds.mds9  5: 'mds9'
2017-10-09 08:56:01.019774 7f1eba44d700  1 mds.mds9  6: '--setuser'
2017-10-09 08:56:01.019775 7f1eba44d700  1 mds.mds9  7: 'ceph'
2017-10-09 08:56:01.019776 7f1eba44d700  1 mds.mds9  8: '--setgroup'
2017-10-09 08:56:01.019778 7f1eba44d700  1 mds.mds9  9: 'ceph'
2017-10-09 08:56:01.019811 7f1eba44d700  1 mds.mds9 respawning with exe
/usr/bin/ceph-mds
2017-10-09 08:56:01.019814 7f1eba44d700  1 mds.mds9  exe_path /proc/self/exe
2017-10-09 08:56:01.046396 7f5ed6090240  0 ceph version 12.2.1
(3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process
(unknown), pid 421
2017-10-09 08:56:01.049516 7f5ed6090240  0 pidfile_write: ignore empty
--pid-file
2017-10-09 08:56:05.162732 7f5ecee32700  1 mds.mds9 handle_mds_map standby
[...]
---snap---

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com