Hi David, thanks for the reply!
Interesting that the package was not installed - it was for us, but the
machines we run the nfs-ganesha servers on are also OSDs, so it might have been
pulled in via ceph-packages for us.
In any case, I'd say this means librados2 as dependency is missing either in
the libcephfs or in nfs-ganesha packages.
Also, good news that things work fine with 12.2.5 - so I hope our upgrade will
also go without bumps ;-).
My experience is sadly only a few months old. We've started with nfs-ganesha
2.5 from the Ceph repos, but hit a bad locking issue, which I also reported to
this list.
After upgrading to 2.6, we did not observe any further hard issues. It seems
that there are sometimes issues with slow locks if processes are running with a
working directory in ceph
and other ceph-fuse clients want to access files in the same directory, but
there are no "deadlock" situations anymore.
In terms of tuning, I did not do anything special yet. I'm running with some
basic NFS / Fileserver kernel tunables (sysctl):
net.core.rmem_max = 12582912
net.core.wmem_max = 12582912
net.ipv4.tcp_rmem = 10240 87380 12582912
net.ipv4.tcp_wmem = 10240 87380 12582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 250000
net.core.default_qdisc = fq_codel
However, I did not do explicit testing of different values, but just followed
general recommendations here.
It seems ACLs and quotas are honoured by the NFS server (as expected, since it
uses libcephfs behind the scenes).
Right now, throughput for bulk data is close to perfect (we manage to saturate
our 1 GBit/s link) and for metadata access it seems close to what ceph-fuse
achieves,
which is sufficient for us.
Cheers and thanks for the feedback,
Oliver
Am 16.05.2018 um 21:06 schrieb David C:
> Hi Oliver
>
> Thanks for following up. I just picked this up again today and it was indeed
> librados2...the package wasn't installed! It's working now, haven't tested
> much but I haven't noticed any problems yet. This is with
> nfs-ganesha-2.6.1-0.1.el7.x86_64, libcephfs2-12.2.5-0.el7.x86_64 and
> librados2-12.2.5-0.el7.x86_64. Thanks for the pointer on that.
>
> I'd be interested to hear your experience with ganesha with cephfs if you're
> happy to share some insights. Any tuning you would recommend?
>
> Thanks,
>
> On Wed, May 16, 2018 at 4:14 PM, Oliver Freyermuth
> <freyerm...@physik.uni-bonn.de <mailto:freyerm...@physik.uni-bonn.de>> wrote:
>
> Hi David,
>
> did you already manage to check your librados2 version and manage to pin
> down the issue?
>
> Cheers,
> Oliver
>
> Am 11.05.2018 um 17:15 schrieb Oliver Freyermuth:
> > Hi David,
> >
> > Am 11.05.2018 um 16:55 schrieb David C:
> >> Hi Oliver
> >>
> >> Thanks for the detailed reponse! I've downgraded my libcephfs2 to
> 12.2.4 and still get a similar error:
> >>
> >> load_fsal :NFS STARTUP :CRIT :Could not dlopen
> module:/usr/lib64/ganesha/libfsalceph.so Error:/lib64/libcephfs.so.2:
> undefined symbol:
> _Z14common_preinitRK18CephInitParameters18code_environment_ti
> >> load_fsal :NFS STARTUP :MAJ :Failed to load module
> (/usr/lib64/ganesha/libfsalceph.so) because: Can not access a needed shared
> library
> >>
> >> I'm on CentOS 7.4, using the following package versions:
> >>
> >> # rpm -qa | grep ganesha
> >> nfs-ganesha-2.6.1-0.1.el7.x86_64
> >> nfs-ganesha-vfs-2.6.1-0.1.el7.x86_64
> >> nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64
> >>
> >> # rpm -qa | grep ceph
> >> libcephfs2-12.2.4-0.el7.x86_64
> >> nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64
> >
> > Mhhhm - that sounds like a messup in the dependencies.
> > The symbol you are missing should be provided by
> > librados2-12.2.4-0.el7.x86_64
> > which contains
> > /usr/lib64/ceph/ceph/libcephfs-common.so.0
> > Do you have a different version of librados2 installed? If so, I wonder
> how yum / rpm allowed that ;-).
> >
> > Thinking again, it might also be (if you indeed have a different
> version there) that this is the cause also for the previous error.
> > If the problematic symbol is indeed not exposed, but can be resolved
> only if both libraries (libcephfs-common and libcephfs) are loaded in unison
> with matching versions,
> > it might be that also 12.2.5 works fine...
> >
> > First thing, in any case, is to checkout which version of librados2 you
> are using ;-).
> >
> > Cheers,
> > Oliver
> >
> >>
> >> I don't have the ceph user space components installed, assuming
> they're not nesscary apart from libcephfs2? Any idea why it's giving me this
> error?
> >>
> >> Thanks,
> >>
> >> On Fri, May 11, 2018 at 2:17 AM, Oliver Freyermuth
> <freyerm...@physik.uni-bonn.de <mailto:freyerm...@physik.uni-bonn.de>
> <mailto:freyerm...@physik.uni-bonn.de
> <mailto:freyerm...@physik.uni-bonn.de>>> wrote:
> >>
> >> Hi David,
> >>
> >> for what it's worth, we are running with nfs-ganesha 2.6.1 from
> Ceph repos on CentOS 7.4 with the following set of versions:
> >> libcephfs2-12.2.4-0.el7.x86_64
> >> nfs-ganesha-2.6.1-0.1.el7.x86_64
> >> nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64
> >> Of course, we plan to upgrade to 12.2.5 soon-ish...
> >>
> >> Am 11.05.2018 um 00:05 schrieb David C:
> >> > Hi All
> >> >
> >> > I'm testing out the nfs-ganesha-2.6.1-0.1.el7.x86_64.rpm package
> from http://download.ceph.com/nfs-ganesha/rpm-V2.6-stable/luminous/x86_64/
> <http://download.ceph.com/nfs-ganesha/rpm-V2.6-stable/luminous/x86_64/>
> <http://download.ceph.com/nfs-ganesha/rpm-V2.6-stable/luminous/x86_64/
> <http://download.ceph.com/nfs-ganesha/rpm-V2.6-stable/luminous/x86_64/>>
> >> >
> >> > It's failing to load /usr/lib64/ganesha/libfsalceph.so
> >> >
> >> > With libcephfs-12.2.1 installed I get the following error in my
> ganesha log:
> >> >
> >> > load_fsal :NFS STARTUP :CRIT :Could not dlopen
> module:/usr/lib64/ganesha/libfsalceph.so Error:
> >> > /usr/lib64/ganesha/libfsalceph.so: undefined symbol:
> ceph_set_deleg_timeout
> >> > load_fsal :NFS STARTUP :MAJ :Failed to load module
> (/usr/lib64/ganesha/libfsalceph.so) because
> >> > : Can not access a needed shared library
> >>
> >> That looks like an ABI incompatibility, probably the nfs-ganesha
> packages should block this libcephfs2-version (and older ones).
> >>
> >> >
> >> >
> >> > With libcephfs-12.2.5 installed I get:
> >> >
> >> > load_fsal :NFS STARTUP :CRIT :Could not dlopen
> module:/usr/lib64/ganesha/libfsalceph.so Error:
> >> > /lib64/libcephfs.so.2: undefined symbol:
> _ZNK5FSMap10parse_roleEN5boost17basic_string_viewIcSt11char_traitsIcEEEP10mds_role_tRSo
> >> > load_fsal :NFS STARTUP :MAJ :Failed to load module
> (/usr/lib64/ganesha/libfsalceph.so) because
> >> > : Can not access a needed shared library
> >>
> >> That looks ugly and makes me fear for our planned 12.2.5-upgrade.
> >> Interestingly, we do not have that symbol on 12.2.4:
> >> # nm -D /lib64/libcephfs.so.2 | grep FSMap
> >> U _ZNK5FSMap10parse_roleERKSsP10mds_role_tRSo
> >> U _ZNK5FSMap13print_summaryEPN4ceph9FormatterEPSo
> >> and NFS-Ganesha works fine.
> >>
> >> Looking at:
> >> https://github.com/ceph/ceph/blob/v12.2.4/src/mds/FSMap.h
> <https://github.com/ceph/ceph/blob/v12.2.4/src/mds/FSMap.h>
> <https://github.com/ceph/ceph/blob/v12.2.4/src/mds/FSMap.h
> <https://github.com/ceph/ceph/blob/v12.2.4/src/mds/FSMap.h>>
> >> versus
> >> https://github.com/ceph/ceph/blob/v12.2.5/src/mds/FSMap.h
> <https://github.com/ceph/ceph/blob/v12.2.5/src/mds/FSMap.h>
> <https://github.com/ceph/ceph/blob/v12.2.5/src/mds/FSMap.h
> <https://github.com/ceph/ceph/blob/v12.2.5/src/mds/FSMap.h>>
> >> it seems this commit:
> >>
> https://github.com/ceph/ceph/commit/7d8b3c1082b6b870710989773f3cd98a472b9a3d
> <https://github.com/ceph/ceph/commit/7d8b3c1082b6b870710989773f3cd98a472b9a3d>
>
> <https://github.com/ceph/ceph/commit/7d8b3c1082b6b870710989773f3cd98a472b9a3d
> <https://github.com/ceph/ceph/commit/7d8b3c1082b6b870710989773f3cd98a472b9a3d>>
> >> changed libcephfs2 ABI.
> >>
> >> I've no idea how that's usually handled and whether ABI breakage
> should occur within point releases (I would not have expected that...).
> >> At least, this means either:
> >> - ABI needs to be reverted to the old state.
> >> - A new NFS Ganesha build is needed. Probably, if this is a common
> thing, builds should be automated and be synchronized to ceph releases,
> >> and old versions should be kept around.
> >>
> >> I'll hold back our update to 12.2.5 until this is resolved, so
> many thanks from my side!
> >>
> >> Let's see who jumps in to resolve it...
> >>
> >> Cheers,
> >> Oliver
> >> >
> >> >
> >> > My cluster is running 12.2.1
> >> >
> >> > All package versions:
> >> >
> >> > nfs-ganesha-2.6.1-0.1.el7.x86_64
> >> > nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64
> >> > libcephfs2-12.2.5-0.el7.x86_64
> >> >
> >> > Can anyone point me in the right direction?
> >> >
> >> > Thanks,
> >> > David
> >> >
> >> >
> >> > _______________________________________________
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
> <mailto:ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>>
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>
> >> >
> >>
> >>
> >>
> >
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> >
>
>
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
