Hi all,
I tried the mds_cache_memory_limit and it doesn’t work.
But unexpectedly, we found drop_cache can work.
We added a "echo 3 > /proc/sys/vm/drop_caches” cron job.
Let it drop cache every 10 minutes and the problem didn’t occur for a whole
night. : )
It seems to work but we can’t
I tested the dirty_ratio and dirty_background_ratio setting on ceph storage
server side.
It didn’t work.
The problem reproduced just now.
Some process stay in D state for over 10 minitues.
These osd requests is about to take several minutes.
Here are some logs：
[Thu Dec 6 17:59:18 2018]
-Original Message-
> From: Li,Ning [mailto:lining916740...@icloud.com]
> Sent: 05 December 2018 11:41
> To: Marc Roos
> Subject: Re: [ceph-users] 【cephfs】cephfs hung when scp/rsync large
> files
>
> Hi Roos,
>
> Thanks for replying.
>
> Our problem’s behavior is
On Thu, Dec 6, 2018 at 2:24 PM Li,Ning wrote:
>
> Hi,
>
> Overload on the storage nodes that run OSDs is normal during the whole copy.
>
> The memory pressure is on the client side. I think it may not work to turn
> the dirty_ratio and dirty_background_ratio to small on the ceph storage
>
Hi,
Overload on the storage nodes that run OSDs is normal during the whole copy.
The memory pressure is on the client side. I think it may not work to turn the
dirty_ratio and dirty_background_ratio to small on the ceph storage server side.
Anyway，I will have a try.
—
Best Regards
Li, Ning
Thanks for your expertise sharing.
We have already been using blue store and SSD for journal actually.
The mds_cache_memory_limit I will have a try.
—
Best Regards
Li, Ning
> On Dec 6, 2018, at 00:44, Zhenshi Zhou wrote:
>
> Hi
>
> I have the same issue a few months ago. One of my
On Wed, Dec 5, 2018 at 2:33 PM NingLi wrote:
>
> Hi all,
>
> We found that some process writing cephfs will hang for a long time (> 120s)
> when uploading(scp/rsync) large files（totally 50G ~ 300G）to the app node's
> cephfs mountpoint.
>
> This problem is not always reproduciable. But when
Hi
I have the same issue a few months ago. One of my client hung on waiting
for a file writing.
And other clients seems not being effected by it. However, if other clients
access to the same
hanged direcotry, it would hang there as well.
My cluster is 12.2.8 and I use kernel client on other
Hi Zheng,
Thanks for replying.
Not the same machine. Our problem is other writing cephfs process will hung
for a long time.
The application node and storage nodes are independent . The cephfs mounts on
application node.
We copy large files to the application node’s cephfs mount point from
Is the cephfs mount on the same machine that run OSD?
On Wed, Dec 5, 2018 at 2:33 PM NingLi wrote:
>
> Hi all,
>
> We found that some process writing cephfs will hang for a long time (> 120s)
> when uploading(scp/rsync) large files（totally 50G ~ 300G）to the app node's
> cephfs mountpoint.
>
: Yilong Ren
Subject: [ceph-users] 【cephfs】cephfs hung when scp/rsync large files
Hi all,
We found that some process writing cephfs will hang for a long time (>
120s) when uploading(scp/rsync) large files（totally 50G ~ 300G）to the
app node's cephfs mountpoint.
This problem is not alw
Hi all,
We found that some process writing cephfs will hang for a long time (> 120s)
when uploading(scp/rsync) large files（totally 50G ~ 300G）to the app node's
cephfs mountpoint.
This problem is not always reproduciable. But when this problem occurs, the
web(nginx) or some other services
12 matches
Mail list logo
