On Wed, Jan 16, 2019 at 7:12 PM Andras Pataki <apat...@flatironinstitute.org> wrote: > > Hi Ilya/Kjetil, > > I've done some debugging and tcpdump-ing to see what the interaction > between the kernel client and the mon looks like. Indeed - > CEPH_MSG_MAX_FRONT defined as 16Mb seems low for the default mon > messages for our cluster (with osd_mon_messages_max at 100). We have > about 3500 osd's, and the kernel advertises itself as older than
This is too big, especially for a fairly large cluster such as yours. The default was reduced to 40 in luminous. Given about 3500 OSDs, you might want to set it to 20 or even 10. > Luminous, so it gets full map updates. The FRONT message size on the > wire I saw was over 24Mb. I'll try setting osd_mon_messages_max to 30 > and do some more testing, but from the debugging it definitely seems > like the issue. > > Is the kernel driver really not up to date to be considered at least a > Luminous client by the mon (i.e. it has some feature really missing)? I > looked at the bits, and the MON seems to want is bit 59 in ceph features > shared by FS_BTIME, FS_CHANGE_ATTR, MSG_ADDR2. Can the kernel client be > used when setting require-min-compat to luminous (either with the 4.19.x > kernel or the Redhat/Centos 7.6 kernel)? Some background here would be > helpful. Yes, the kernel client is missing support for that feature bit, however 4.13+ and RHEL 7.5+ _can_ be used with require-min-compat-client set to luminous. See http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/027002.html Thanks, Ilya _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
