Hi, I find an old server which mounted cephfs and has the debug files. # cat osdc REQUESTS 0 homeless 0 LINGER REQUESTS BACKOFFS # cat monc have monmap 2 want 3+ have osdmap 3507 have fsmap.user 0 have mdsmap 55 want 56+ fs_cluster_id -1 # cat mdsc 194 mds0 getattr #10000036ae3
What does it mean? Zhenshi Zhou <[email protected]> 于2018年8月8日周三 下午1:58写道： > I restarted the client server so that there's no file in that directory. I > will take care of it if the client hangs next time. > > Thanks > > Yan, Zheng <[email protected]> 于2018年8月8日周三 上午11:23写道： > >> On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou <[email protected]> wrote: >> > >> > Hi, >> > I check all my ceph servers and they are not mount cephfs on each of >> them(maybe I umount after testing). As a result, the cluster didn't >> encounter a memory deadlock. Besides, I check the monitoring system and the >> memory and cpu usage were at common level while the clients hung. >> > Back to my question, there must be something else cause the client hang. >> > >> >> Check if there are hang requests in >> /sys/kernel/debug/ceph/xxxx/{osdc,mdsc}, >> >> > Zhenshi Zhou <[email protected]> 于2018年8月8日周三 上午4:16写道： >> >> >> >> Hi, I'm not sure if it just mounts the cephfs without using or doing >> any operation within the mounted directory would be affected by flushing >> cache. I mounted cephfs on osd servers only for testing and then left it >> there. Anyway I will umount it. >> >> >> >> Thanks >> >> >> >> John Spray <[email protected]>于2018年8月8日 周三03:37写道： >> >>> >> >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier <[email protected]> >> wrote: >> >>> > >> >>> > This is the first I am hearing about this as well. >> >>> >> >>> This is not a Ceph-specific thing -- it can also affect similar >> >>> systems like Lustre. >> >>> >> >>> The classic case is when under some memory pressure, the kernel tries >> >>> to free memory by flushing the client's page cache, but doing the >> >>> flush means allocating more memory on the server, making the memory >> >>> pressure worse, until the whole thing just seizes up. >> >>> >> >>> John >> >>> >> >>> > Granted, I am using ceph-fuse rather than the kernel client at this >> point, but that isn’t etched in stone. >> >>> > >> >>> > Curious if there is more to share. >> >>> > >> >>> > Reed >> >>> > >> >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima < >> [email protected]> wrote: >> >>> > >> >>> > >> >>> > Yan, Zheng <[email protected]> 于2018年8月7日周二 下午7:51写道： >> >>> >> >> >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou <[email protected]> >> wrote: >> >>> >> this can cause memory deadlock. you should avoid doing this >> >>> >> >> >>> >> > Yan, Zheng <[email protected]>于2018年8月7日 周二19:12写道： >> >>> >> >> >> >>> >> >> did you mount cephfs on the same machines that run ceph-osd? >> >>> >> >> >> >>> > >> >>> > >> >>> > I didn't know about this. I run this setup in production. :P >> >>> > >> >>> > Regards, >> >>> > >> >>> > Webert Lima >> >>> > DevOps Engineer at MAV Tecnologia >> >>> > Belo Horizonte - Brasil >> >>> > IRC NICK - WebertRLZ >> >>> > >> >>> > _______________________________________________ >> >>> > ceph-users mailing list >> >>> > [email protected] >> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>> > >> >>> > >> >>> > _______________________________________________ >> >>> > ceph-users mailing list >> >>> > [email protected] >> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>> _______________________________________________ >> >>> ceph-users mailing list >> >>> [email protected] >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> > _______________________________________________ >> > ceph-users mailing list >> > [email protected] >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
