Re: [ceph-users] Feedback wanted: health warning when standby MDS dies?
Hi, I would be interested in this case when a mds in standby-replay fails. Thanks On Wed, Oct 19, 2016 at 4:06 PM, Scottix wrote: > I would take the analogy of a Raid scenario. Basically a standby is > considered like a spare drive. If that spare drive goes down. It is good to > know about the event, but it does in no way indicate a degraded system, > everything keeps running at top speed. > > If you had multi active MDS and one goes down then I would say that is a > degraded system, but still waiting for that feature. > > > On Tue, Oct 18, 2016 at 10:18 AM Goncalo Borges < > goncalo.bor...@sydney.edu.au> wrote: > >> Hi John. >> >> That would be good. >> >> In our case we are just picking that up simply through nagios and some >> fancy scripts parsing the dump of the MDS maps. >> >> Cheers >> Goncalo >> >> From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of John >> Spray [jsp...@redhat.com] >> Sent: 18 October 2016 22:46 >> To: ceph-users >> Subject: [ceph-users] Feedback wanted: health warning when standby MDS >> dies? >> >> Hi all, >> >> Someone asked me today how to get a list of down MDS daemons, and I >> explained that currently the MDS simply forgets about any standby that >> stops sending beacons. That got me thinking about the case where a >> standby dies while the active MDS remains up -- the cluster has gone >> into a non-highly-available state, but we are not giving the admin any >> indication. >> >> I've suggested a solution here: >> http://tracker.ceph.com/issues/17604 >> >> This is probably going to be a bit of a subjective thing in terms of >> whether people find it useful or find it to be annoying noise, so I'd >> be interested in feedback from people currently running cephfs. >> >> Cheers, >> John >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Feedback wanted: health warning when standby MDS dies?
I would take the analogy of a Raid scenario. Basically a standby is considered like a spare drive. If that spare drive goes down. It is good to know about the event, but it does in no way indicate a degraded system, everything keeps running at top speed. If you had multi active MDS and one goes down then I would say that is a degraded system, but still waiting for that feature. On Tue, Oct 18, 2016 at 10:18 AM Goncalo Borges < goncalo.bor...@sydney.edu.au> wrote: > Hi John. > > That would be good. > > In our case we are just picking that up simply through nagios and some > fancy scripts parsing the dump of the MDS maps. > > Cheers > Goncalo > > From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of John > Spray [jsp...@redhat.com] > Sent: 18 October 2016 22:46 > To: ceph-users > Subject: [ceph-users] Feedback wanted: health warning when standby MDS > dies? > > Hi all, > > Someone asked me today how to get a list of down MDS daemons, and I > explained that currently the MDS simply forgets about any standby that > stops sending beacons. That got me thinking about the case where a > standby dies while the active MDS remains up -- the cluster has gone > into a non-highly-available state, but we are not giving the admin any > indication. > > I've suggested a solution here: > http://tracker.ceph.com/issues/17604 > > This is probably going to be a bit of a subjective thing in terms of > whether people find it useful or find it to be annoying noise, so I'd > be interested in feedback from people currently running cephfs. > > Cheers, > John > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Feedback wanted: health warning when standby MDS dies?
Hi John. That would be good. In our case we are just picking that up simply through nagios and some fancy scripts parsing the dump of the MDS maps. Cheers Goncalo From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of John Spray [jsp...@redhat.com] Sent: 18 October 2016 22:46 To: ceph-users Subject: [ceph-users] Feedback wanted: health warning when standby MDS dies? Hi all, Someone asked me today how to get a list of down MDS daemons, and I explained that currently the MDS simply forgets about any standby that stops sending beacons. That got me thinking about the case where a standby dies while the active MDS remains up -- the cluster has gone into a non-highly-available state, but we are not giving the admin any indication. I've suggested a solution here: http://tracker.ceph.com/issues/17604 This is probably going to be a bit of a subjective thing in terms of whether people find it useful or find it to be annoying noise, so I'd be interested in feedback from people currently running cephfs. Cheers, John ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Feedback wanted: health warning when standby MDS dies?
On 2016-10-18T12:46:48, John Spray wrote: > I've suggested a solution here: > http://tracker.ceph.com/issues/17604 > > This is probably going to be a bit of a subjective thing in terms of > whether people find it useful or find it to be annoying noise, so I'd > be interested in feedback from people currently running cephfs. I'm not adding much new value here, but, yes. Being warned that we're in a degraded mode where the next failure might cause an outage would be very useful. Maybe that's even worth flagging as a separate health level. -- SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Feedback wanted: health warning when standby MDS dies?
+1 to this, it would be useful On Tue, Oct 18, 2016 at 8:31 AM, Wido den Hollander wrote: > >> Op 18 oktober 2016 om 14:06 schreef Dan van der Ster : >> >> >> +1 I would find this warning useful. >> > > +1 Probably make it configurable, say, you want at least X standby MDS to be > available before WARN. But in general, yes, please! > > Wido > >> >> >> On Tue, Oct 18, 2016 at 1:46 PM, John Spray wrote: >> > Hi all, >> > >> > Someone asked me today how to get a list of down MDS daemons, and I >> > explained that currently the MDS simply forgets about any standby that >> > stops sending beacons. That got me thinking about the case where a >> > standby dies while the active MDS remains up -- the cluster has gone >> > into a non-highly-available state, but we are not giving the admin any >> > indication. >> > >> > I've suggested a solution here: >> > http://tracker.ceph.com/issues/17604 >> > >> > This is probably going to be a bit of a subjective thing in terms of >> > whether people find it useful or find it to be annoying noise, so I'd >> > be interested in feedback from people currently running cephfs. >> > >> > Cheers, >> > John >> > ___ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Feedback wanted: health warning when standby MDS dies?
> Op 18 oktober 2016 om 14:06 schreef Dan van der Ster : > > > +1 I would find this warning useful. > +1 Probably make it configurable, say, you want at least X standby MDS to be available before WARN. But in general, yes, please! Wido > > > On Tue, Oct 18, 2016 at 1:46 PM, John Spray wrote: > > Hi all, > > > > Someone asked me today how to get a list of down MDS daemons, and I > > explained that currently the MDS simply forgets about any standby that > > stops sending beacons. That got me thinking about the case where a > > standby dies while the active MDS remains up -- the cluster has gone > > into a non-highly-available state, but we are not giving the admin any > > indication. > > > > I've suggested a solution here: > > http://tracker.ceph.com/issues/17604 > > > > This is probably going to be a bit of a subjective thing in terms of > > whether people find it useful or find it to be annoying noise, so I'd > > be interested in feedback from people currently running cephfs. > > > > Cheers, > > John > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Feedback wanted: health warning when standby MDS dies?
+1 I would find this warning useful. On Tue, Oct 18, 2016 at 1:46 PM, John Spray wrote: > Hi all, > > Someone asked me today how to get a list of down MDS daemons, and I > explained that currently the MDS simply forgets about any standby that > stops sending beacons. That got me thinking about the case where a > standby dies while the active MDS remains up -- the cluster has gone > into a non-highly-available state, but we are not giving the admin any > indication. > > I've suggested a solution here: > http://tracker.ceph.com/issues/17604 > > This is probably going to be a bit of a subjective thing in terms of > whether people find it useful or find it to be annoying noise, so I'd > be interested in feedback from people currently running cephfs. > > Cheers, > John > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Feedback wanted: health warning when standby MDS dies?
Hi all, Someone asked me today how to get a list of down MDS daemons, and I explained that currently the MDS simply forgets about any standby that stops sending beacons. That got me thinking about the case where a standby dies while the active MDS remains up -- the cluster has gone into a non-highly-available state, but we are not giving the admin any indication. I've suggested a solution here: http://tracker.ceph.com/issues/17604 This is probably going to be a bit of a subjective thing in terms of whether people find it useful or find it to be annoying noise, so I'd be interested in feedback from people currently running cephfs. Cheers, John ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com