[ceph-users] osdmap several thousand epochs behind latest
All,
Had a fairly substantial network interruption that knocked out about
~270 osds:
health HEALTH_ERR
[...]
273/384 in osds are down
noup,nodown,noout flag(s) set
monmap e2: 3 mons at
{cephmon-0=10.10.6.0:6789/0,cephmon-1=10.10.6.1:6789/0,cephmon-2=10.10.6.2:6789/0}
election epoch 138, quorum 0,1,2
cephmon-0,cephmon-1,cephmon-2
mgr no daemons active
osdmap e37718: 384 osds: 111 up, 384 in; 16764 remapped pgs
flags
noup,nodown,noout,sortbitwise,require_jewel_osds,require_kraken_osds
We've had network interruptions before, and normally OSDs come back on
their own, or do so with a service restart. This time, no such luck
(I'm guessing the scale was just too much). After a few hours of trying
to figure out why OSD services were running on the hosts (according to
systemd) but marked 'down' in ceph osd tree, I found this thread:
http://ceph-devel.vger.kernel.narkive.com/ftEN7TOU/70-osd-are-down-and-not-coming-up
which appears to perfectly describe the scenario (high CPU usage, osdmap
way out of sync, etc.)
I've taken the steps outlined and set the appropriate flags and am
monitoring the 'catch up' progress of the OSDs. The OSD farthest behind
is about 5000 epochs out of sync, so I assume it will be a few hours
before I see CPU usage level out.
Once the OSDs are caught up, are there any other steps I should take
before 'ceph osd unset noup' (or anything to do after)?
Thanks in advance,
--
v/r
Chris Apsey
bitskr...@bitskrieg.net
https://www.bitskrieg.net
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to Rebuild libvirt + qemu packages with Ceph support on Debian 9.0 Stretch
On 07/08/2017 12:49 AM, Luescher Claude wrote: > Hello List, > > It would be nice if somebody would write an up2date tutorial for this > since Stretch is now the official distro or provide packages with ceph > support on a separate repo. > > I have to say that it's a big FAIL for them that this important support > functionality is not already included in the stock debian package! > > I believe kvm itself has the support but not libvirt. These are the > stock packages: I suspect that you may need the package 'qemu-block-extra' installed. It has a dependency on such things as librados2, librbd1 (for Ceph) and glusterfs-common, libiscsi7, etc. From the package description: "This package provides extra block device backend modules for qemu-system emulation and qemu-img from qemu-utils package, which are rarely used and has extra dependencies." -- David Clarke Systems Architect Catalyst IT signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Stealth Jewel release?
Hello, so this morning I was greeted with the availability of 10.2.8 for both Jessie and Stretch (much appreciated), but w/o any announcement here or updated release notes on the website, etc. Any reason other "Friday" (US time) for this? Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Communications ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Stealth Jewel release?
On 17-07-10 08:29, Christian Balzer wrote: Hello, so this morning I was greeted with the availability of 10.2.8 for both Jessie and Stretch (much appreciated), but w/o any announcement here or updated release notes on the website, etc. Any reason other "Friday" (US time) for this? Christian My guess is that they didn't have time to announce it yet. Maybe pkgs were not ready yet on friday? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
