Marc,
Thanks.
The mgr log spam occurs even without dashboard module enabled. I never
checked the ceph mgr log before because the ceph cluster is always healthy.
Based on the ceph mgr logs in syslog, the spam occurred long before and
after I enabled the dashboard module.
# ceph -s
> cluster:
> id: xxx
> health: HEALTH_OK
>
> services:
> mon: 3 daemons, quorum mon1,mon2,mon3
> mgr: mon1(active), standbys: mon2, mon3
> mds: cephfs-1/1/1 up {0=mds1=up:active}, 2 up:standby
> osd: 14 osds: 14 up, 14 in
> rgw: 3 daemons active
>
> data:
> pools: 10 pools, 248 pgs
> objects: 546k objects, 2119 GB
> usage: 6377 GB used, 6661 GB / 13039 GB avail
> pgs: 248 active+clean
>
> io:
> client: 25233 B/s rd, 1409 kB/s wr, 6 op/s rd, 59 op/s wr
My ceph mgr log is spam with following log every second. This happens on 2
separate Ceph 12.2.4 clusters.
# less +F /var/log/ceph/ceph-mgr.mon1.log
...
2018-04-20 06:21:18.782861 7fca238ff700 1 mgr send_beacon active
> 2018-04-20 06:21:19.050671 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf897d1c00 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:19.051047 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> 2018-04-20 06:21:20.050889 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf897eac00 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:20.051351 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> 2018-04-20 06:21:20.784455 7fca238ff700 1 mgr send_beacon active
> 2018-04-20 06:21:21.050968 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf897d0d00 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:21.051441 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> 2018-04-20 06:21:22.051254 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf897ec100 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:22.051704 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> 2018-04-20 06:21:22.786656 7fca238ff700 1 mgr send_beacon active
> 2018-04-20 06:21:23.051235 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf897d0400 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:23.051712 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> 2018-04-20 06:21:24.051353 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf897e6000 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:24.051971 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> 2018-04-20 06:21:24.788228 7fca238ff700 1 mgr send_beacon active
> 2018-04-20 06:21:25.051642 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf897d1900 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:25.052182 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> 2018-04-20 06:21:26.051641 7fca14809700 0 ms_deliver_dispatch: unhandled
> message 0x55bf89835600 mgrreport(mds.mds1 +24-0 packed 214) v5 from mds.0
> 10.100.100.114:6800/4132681434
> 2018-04-20 06:21:26.052169 7fca25102700 1 mgr finish mon failed to return
> metadata for mds.mds1: (2) No such file or directory
> ...
Kind regards,
Charles Alva
Sent from Gmail Mobile
On Fri, Apr 20, 2018 at 10:57 AM, Marc Roos <[email protected]>
wrote:
>
> Hi Charles,
>
> I am more or less responding to your syslog issue. I don’t have the
> experience on cephfs to give you a reliable advice. So lets wait for the
> experts to reply. But I guess you have to give a little more background
> info, like
>
> This happened to running cluster, you didn’t apply any changes to?
> Looks like your dashboard issue is not related to "1 mgr finish mon
> failed to return metadata for mds.mds1"
>
>
> -----Original Message-----
> From: Charles Alva [mailto:[email protected]]
> Sent: vrijdag 20 april 2018 10:33
> To: Marc Roos
> Cc: ceph-users
> Subject: Re: [ceph-users] Ceph 12.2.4 MGR spams syslog with "mon failed
> to return metadata for mds"
>
> Hi Marc,
>
> I'm using CephFS and mgr could not get the metadata of the mds. I
> enabled the dashboard module and everytime I visit the ceph filesystem
> page, it got internal error 500.
>
> Kind regards,
>
> Charles Alva
> Sent from Gmail Mobile
>
>
> On Fri, Apr 20, 2018 at 9:24 AM, Marc Roos <[email protected]>
> wrote:
>
>
>
> Remote syslog server, and buffering writes to the log?
>
>
> Actually this is another argument to fix logging to syslog a bit,
> because the default syslog is also be set to throttle and group
> the
>
> messages like:
>
> Mar 9 17:59:35 db1 influxd: last message repeated 132 times
>
>
> https://www.mail-archive.com/[email protected]/
> msg45025.htm
> l
> <https://www.mail-archive.com/[email protected]/msg45025.html>
>
>
>
>
>
> -----Original Message-----
> From: Charles Alva [mailto:[email protected]]
> Sent: vrijdag 20 april 2018 8:08
> To: [email protected]
> Subject: [ceph-users] Ceph 12.2.4 MGR spams syslog with "mon
> failed
> to
> return metadata for mds"
>
> Hi All,
>
> Just noticed on 2 Ceph Luminous 12.2.4 clusters, Ceph mgr spams
> the
>
> syslog with lots of "mon failed to return metadata for mds" every
> second.
>
> ```
> 2018-04-20 06:06:03.951412 7fca238ff700 1 mgr send_beacon active
> 2018-04-20 06:06:04.934477 7fca14809700 0 ms_deliver_dispatch:
> unhandled message 0x55bf897f0a00 mgrreport(mds.mds1 +24-0 packed
> 214) v5
> from mds.0 10.100.100.114:6800/4132681434 2018-04-20
> 06:06:04.934937
> 7fca25102700 1 mgr finish mon failed to return metadata for
> mds.mds1:
> (2) No such file or directory ```
>
> How to fix this issue? or disable it completely to reduce disk IO
> and
> increase SSD life span?
>
>
>
> Kind regards,
>
> Charles Alva
> Sent from Gmail Mobile
>
>
>
>
>
>
>
>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
