On Fri, 2014-06-06 at 17:40 +0400, Ilya Dryomov wrote: > On Fri, Jun 6, 2014 at 4:34 PM, Kenneth Waegeman > <[email protected]> wrote: > > > > ----- Message from Igor Krstic <[email protected]> --------- > > Date: Fri, 06 Jun 2014 13:23:19 +0200 > > From: Igor Krstic <[email protected]> > > Subject: Re: [ceph-users] question about feature set mismatch > > To: Ilya Dryomov <[email protected]> > > Cc: [email protected] > > > > > > > >> On Fri, 2014-06-06 at 11:51 +0400, Ilya Dryomov wrote: > >>> > >>> On Thu, Jun 5, 2014 at 10:38 PM, Igor Krstic <[email protected]> > >>> wrote: > >>> > Hello, > >>> > > >>> > dmesg: > >>> > [ 690.181780] libceph: mon1 192.168.214.102:6789 feature set mismatch, > >>> > my > >>> > 4a042a42 < server's 504a042a42, missing 5000000000 > >>> > [ 690.181907] libceph: mon1 192.168.214.102:6789 socket error on read > >>> > [ 700.190342] libceph: mon0 192.168.214.101:6789 feature set mismatch, > >>> > my > >>> > 4a042a42 < server's 504a042a42, missing 5000000000 > >>> > [ 700.190481] libceph: mon0 192.168.214.101:6789 socket error on read > >>> > [ 710.194499] libceph: mon1 192.168.214.102:6789 feature set mismatch, > >>> > my > >>> > 4a042a42 < server's 504a042a42, missing 5000000000 > >>> > [ 710.194633] libceph: mon1 192.168.214.102:6789 socket error on read > >>> > [ 720.201226] libceph: mon1 192.168.214.102:6789 feature set mismatch, > >>> > my > >>> > 4a042a42 < server's 504a042a42, missing 5000000000 > >>> > [ 720.201482] libceph: mon1 192.168.214.102:6789 socket error on read > >>> > > >>> > 5000000000 should be: > >>> > CEPH_FEATURE_CRUSH_V2 36 1000000000 > >>> > and > >>> > CEPH_FEATURE_OSD_ERASURE_CODES 38 4000000000 > >>> > CEPH_FEATURE_OSD_TMAP2OMAP 38* 4000000000 > >>> > > >>> > That is happening on two separate boxes that are just my nfs and block > >>> > gateways (they are not osd/mon/mds). So I just need on them something > >>> > like: > >>> > sudo rbd map share2 > >>> > sudo mount -t xfs /dev/rbd1 /mnt/share2 > >>> > > >>> > On ceph cluster and on those two separate boxes: > >>> > ~$ ceph -v > >>> > ceph version 0.80.1 > >>> > > >>> > What could be the problem? > >>> > >>> Which kernel version are you running? Do you have any erasure coded > >>> pools? > >>> > >>> Thanks, > >>> > >>> Ilya > >> > >> ~$ uname -a > >> Linux ceph-gw1 3.13.0-24-generic #47~precise2-Ubuntu SMP Fri May 2 > >> 23:30:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > >> Yes, one of the pools is erasure coded pool but the only thing I use on > >> that box is rbd and rbd pool is not ec pool. It is replicated pool. I to > >> not touch ec pool from there. Or at least, I believe so :) > > > > > > Well, I saw something similar with CephFS: I didn't touch the pools in use > > by cephfs, but I created another pool with Erasure Code, and the ceph client > > (kernel 3.13, but not enough for EC) also stopped working with 'feature set > > mismatch'. Thus I guess the clients can't read the crushmap anymore when > > there is a 'erasure' mentioning in it:) > > Unfortunately that's true. If there are any erasure code pools in the > cluster, kernel clients (both krbd and kcephfs) won't work. The only > way it will work is if you remove all erasure coded pools. > > CRUSH_V2 is also not present in 3.13. You'll have to uprade to 3.14. > Alternatively, CRUSH_V2 can be disabled, but I can't tell you how off > the top of my head. The fundamental problem is that you are running > latest userspace, and the defaults it ships with are incompatible with > older kernels. > > Thanks, > > Ilya
Thanks. Upgrade to 3.14 solved CRUSH_V2. Regarding krbd and kcephfs... If that is the case, that is something that should be addressed in documentation on ceph.com more clearly. There is info that CRUSH_TUNABLES3 (chooseleaf_vary_r) requires "Linux kernel version v3.15 or later (for the file system and RBD kernel clients)" but nothing else. Only now that I have your information I was able to find https://lkml.org/lkml/2014/4/7/257 Anyway... What I want to test is SSD pool as cache pool in front of ec pool. Is there some way to update krbd manually (from github?) or I need to wait 3.15 for this? Thanks, Igor _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
