You can only try to remount the cephs dir. It will probably not work,
giving you I/O Errors, so the fallback would be to use a fuse-mount.
If I recall correctly you could do a lazy umount on the current dir (umount
-fl /mountdir) and remount it using the FUSE client.
it will work for new sessions but the currently hanging ones will still be
hanging.
with fuse you'll only be able to mount cephfs root dir, so if you have
multiple directories, you'll need to:
- mount root cephfs dir in another directory
- mount each subdir (after root mounted) to the desired directory via bind
mount.
Regards,
Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*
On Wed, Aug 8, 2018 at 11:46 AM Zhenshi Zhou <deader...@gmail.com> wrote:
> Hi,
> Is there any other way excpet rebooting the server when the client hangs?
> If the server is in production environment, I can't restart it everytime.
>
> Webert de Souza Lima <webert.b...@gmail.com> 于2018年8月8日周三 下午10:33写道：
>
>> Hi Zhenshi,
>>
>> if you still have the client mount hanging but no session is connected,
>> you probably have some PID waiting with blocked IO from cephfs mount.
>> I face that now and then and the only solution is to reboot the server,
>> as you won't be able to kill a process with pending IO.
>>
>> Regards,
>>
>> Webert Lima
>> DevOps Engineer at MAV Tecnologia
>> *Belo Horizonte - Brasil*
>> *IRC NICK - WebertRLZ*
>>
>>
>> On Wed, Aug 8, 2018 at 11:17 AM Zhenshi Zhou <deader...@gmail.com> wrote:
>>
>>> Hi Webert,
>>> That command shows the current sessions, whereas the server which I get
>>> the files(osdc,mdsc,monc) disconnect for a long time.
>>> So I cannot get useful infomation from the command you provide.
>>>
>>> Thanks
>>>
>>> Webert de Souza Lima <webert.b...@gmail.com> 于2018年8月8日周三 下午10:10写道：
>>>
>>>> You could also see open sessions at the MDS server by issuing `ceph
>>>> daemon mds.XX session ls`
>>>>
>>>> Regards,
>>>>
>>>> Webert Lima
>>>> DevOps Engineer at MAV Tecnologia
>>>> *Belo Horizonte - Brasil*
>>>> *IRC NICK - WebertRLZ*
>>>>
>>>>
>>>> On Wed, Aug 8, 2018 at 5:08 AM Zhenshi Zhou <deader...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, I find an old server which mounted cephfs and has the debug files.
>>>>> # cat osdc
>>>>> REQUESTS 0 homeless 0
>>>>> LINGER REQUESTS
>>>>> BACKOFFS
>>>>> # cat monc
>>>>> have monmap 2 want 3+
>>>>> have osdmap 3507
>>>>> have fsmap.user 0
>>>>> have mdsmap 55 want 56+
>>>>> fs_cluster_id -1
>>>>> # cat mdsc
>>>>> 194 mds0 getattr #10000036ae3
>>>>>
>>>>> What does it mean?
>>>>>
>>>>> Zhenshi Zhou <deader...@gmail.com> 于2018年8月8日周三 下午1:58写道：
>>>>>
>>>>>> I restarted the client server so that there's no file in that
>>>>>> directory. I will take care of it if the client hangs next time.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Yan, Zheng <uker...@gmail.com> 于2018年8月8日周三 上午11:23写道：
>>>>>>
>>>>>>> On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou <deader...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Hi,
>>>>>>> > I check all my ceph servers and they are not mount cephfs on each
>>>>>>> of them(maybe I umount after testing). As a result, the cluster didn't
>>>>>>> encounter a memory deadlock. Besides, I check the monitoring system and
>>>>>>> the
>>>>>>> memory and cpu usage were at common level while the clients hung.
>>>>>>> > Back to my question, there must be something else cause the client
>>>>>>> hang.
>>>>>>> >
>>>>>>>
>>>>>>> Check if there are hang requests in
>>>>>>> /sys/kernel/debug/ceph/xxxx/{osdc,mdsc},
>>>>>>>
>>>>>>> > Zhenshi Zhou <deader...@gmail.com> 于2018年8月8日周三 上午4:16写道：
>>>>>>> >>
>>>>>>> >> Hi, I'm not sure if it just mounts the cephfs without using or
>>>>>>> doing any operation within the mounted directory would be affected by
>>>>>>> flushing cache. I mounted cephfs on osd servers only for testing and
>>>>>>> then
>>>>>>> left it there. Anyway I will umount it.
>>>>>>> >>
>>>>>>> >> Thanks
>>>>>>> >>
>>>>>>> >> John Spray <jsp...@redhat.com>于2018年8月8日 周三03:37写道：
>>>>>>> >>>
>>>>>>> >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier <reed.d...@focusvq.com>
>>>>>>> wrote:
>>>>>>> >>> >
>>>>>>> >>> > This is the first I am hearing about this as well.
>>>>>>> >>>
>>>>>>> >>> This is not a Ceph-specific thing -- it can also affect similar
>>>>>>> >>> systems like Lustre.
>>>>>>> >>>
>>>>>>> >>> The classic case is when under some memory pressure, the kernel
>>>>>>> tries
>>>>>>> >>> to free memory by flushing the client's page cache, but doing the
>>>>>>> >>> flush means allocating more memory on the server, making the
>>>>>>> memory
>>>>>>> >>> pressure worse, until the whole thing just seizes up.
>>>>>>> >>>
>>>>>>> >>> John
>>>>>>> >>>
>>>>>>> >>> > Granted, I am using ceph-fuse rather than the kernel client at
>>>>>>> this point, but that isn’t etched in stone.
>>>>>>> >>> >
>>>>>>> >>> > Curious if there is more to share.
>>>>>>> >>> >
>>>>>>> >>> > Reed
>>>>>>> >>> >
>>>>>>> >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima <
>>>>>>> webert.b...@gmail.com> wrote:
>>>>>>> >>> >
>>>>>>> >>> >
>>>>>>> >>> > Yan, Zheng <uker...@gmail.com> 于2018年8月7日周二 下午7:51写道：
>>>>>>> >>> >>
>>>>>>> >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou <
>>>>>>> deader...@gmail.com> wrote:
>>>>>>> >>> >> this can cause memory deadlock. you should avoid doing this
>>>>>>> >>> >>
>>>>>>> >>> >> > Yan, Zheng <uker...@gmail.com>于2018年8月7日 周二19:12写道：
>>>>>>> >>> >> >>
>>>>>>> >>> >> >> did you mount cephfs on the same machines that run
>>>>>>> ceph-osd?
>>>>>>> >>> >> >>
>>>>>>> >>> >
>>>>>>> >>> >
>>>>>>> >>> > I didn't know about this. I run this setup in production. :P
>>>>>>> >>> >
>>>>>>> >>> > Regards,
>>>>>>> >>> >
>>>>>>> >>> > Webert Lima
>>>>>>> >>> > DevOps Engineer at MAV Tecnologia
>>>>>>> >>> > Belo Horizonte - Brasil
>>>>>>> >>> > IRC NICK - WebertRLZ
>>>>>>> >>> >
>>>>>>> >>> > _______________________________________________
>>>>>>> >>> > ceph-users mailing list
>>>>>>> >>> > ceph-users@lists.ceph.com
>>>>>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>> >>> >
>>>>>>> >>> >
>>>>>>> >>> > _______________________________________________
>>>>>>> >>> > ceph-users mailing list
>>>>>>> >>> > ceph-users@lists.ceph.com
>>>>>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>> >>> _______________________________________________
>>>>>>> >>> ceph-users mailing list
>>>>>>> >>> ceph-users@lists.ceph.com
>>>>>>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > ceph-users mailing list
>>>>>>> > ceph-users@lists.ceph.com
>>>>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>
>>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@lists.ceph.com
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>>>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
