We don't use hardlink.
I reduced the mds_cache_size from 1000 to 200.
After that, the num_strays reduce to about 100k
The cluster is normal now. I think there is some bug about it.
Anyway, thanks for your reply!
-邮件原件-
发件人: Cary [mailto:dynamic.c...@gmail.com]
发送时间: 2017年12月26日 14:08
收件人: 周 威 <cho...@msn.cn>
抄送: ceph-users@lists.ceph.com
主题: Re: 答复: [ceph-users] Can't delete file in cephfs with "No space left on
device"
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ceph.com%2Fpipermail%2Fceph-users-ceph.com%2F2016-October%2F013646.html=02%7C01%7C%7C571893beb17d4d643a8e08d54c26fbed%7C84df9e7fe9f640afb435%7C1%7C0%7C636498652669125646=2Y9JT%2BoksSglveXPttiR7y3nwAmADhLwxTUoH4lxQAI%3D=0
On Tue, Dec 26, 2017 at 6:07 AM, Cary <dynamic.c...@gmail.com> wrote:
> Are you using hardlinks in cephfs?
>
>
> On Tue, Dec 26, 2017 at 3:42 AM, 周 威 <cho...@msn.cn> wrote:
>> The out put of ceph osd df
>>
>>
>>
>> ID WEIGHT REWEIGHT SIZE USEAVAIL %USE VAR PGS
>>
>> 0 1.62650 1.0 1665G 1279G 386G 76.82 1.05 343
>>
>> 1 1.62650 1.0 1665G 1148G 516G 68.97 0.94 336
>>
>> 2 1.62650 1.0 1665G 1253G 411G 75.27 1.03 325
>>
>> 3 1.62650 1.0 1665G 1192G 472G 71.60 0.98 325
>>
>> 4 1.62650 1.0 1665G 1205G 460G 72.35 0.99 341
>>
>> 5 1.62650 1.0 1665G 1381G 283G 82.95 1.13 364
>>
>> 6 1.62650 1.0 1665G 1069G 595G 64.22 0.88 322
>>
>> 7 1.62650 1.0 1665G 1222G 443G 73.38 1.00 337
>>
>> 8 1.62650 1.0 1665G 1120G 544G 67.29 0.92 312
>>
>> 9 1.62650 1.0 1665G 1166G 498G 70.04 0.96 336
>>
>> 10 1.62650 1.0 1665G 1254G 411G 75.31 1.03 348
>>
>> 11 1.62650 1.0 1665G 1352G 313G 81.19 1.11 341
>>
>> 12 1.62650 1.0 1665G 1174G 490G 70.52 0.96 328
>>
>> 13 1.62650 1.0 1665G 1281G 383G 76.95 1.05 345
>>
>> 14 1.62650 1.0 1665G 1147G 518G 68.88 0.94 339
>>
>> 15 1.62650 1.0 1665G 1236G 429G 74.24 1.01 334
>>
>> 20 1.62650 1.0 1665G 1166G 499G 70.03 0.96 325
>>
>> 21 1.62650 1.0 1665G 1371G 293G 82.35 1.13 377
>>
>> 22 1.62650 1.0 1665G 1110G 555G 66.67 0.91 341
>>
>> 23 1.62650 1.0 1665G 1221G 443G 73.36 1.00 327
>>
>> 16 1.62650 1.0 1665G 1354G 310G 81.34 1.11 352
>>
>> 17 1.62650 1.0 1665G 1250G 415G 75.06 1.03 341
>>
>> 18 1.62650 1.0 1665G 1179G 486G 70.80 0.97 316
>>
>> 19 1.62650 1.0 1665G 1236G 428G 74.26 1.01 333
>>
>> 24 1.62650 1.0 1665G 1146G 518G 68.86 0.94 325
>>
>> 25 1.62650 1.0 1665G 1033G 632G 62.02 0.85 309
>>
>> 26 1.62650 1.0 1665G 1234G 431G 74.11 1.01 334
>>
>> 27 1.62650 1.0 1665G 1342G 322G 80.62 1.10 352
>>
>> TOTAL 46635G 34135G 12500G 73.20
>>
>> MIN/MAX VAR: 0.85/1.13 STDDEV: 5.28
>>
>>
>>
>> 发件人: Cary [mailto:dynamic.c...@gmail.com]
>> 发送时间: 2017年12月26日 11:40
>> 收件人: ? ? <cho...@msn.cn>
>> 抄送: ceph-users@lists.ceph.com
>> 主题: Re: [ceph-users] Can't delete file in cephfs with "No space left
>> on device"
>>
>>
>>
>> Could you post the output of “ceph osd df”?
>>
>>
>> On Dec 25, 2017, at 19:46, ? ? <cho...@msn.cn> wrote:
>>
>> Hi all:
>>
>>
>>
>> Ceph version:
>> ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
>>
>>
>>
>> Ceph df:
>>
>> GLOBAL:
>>
>> SIZE AVAIL RAW USED %RAW USED
>>
>> 46635G 12500G 34135G 73.19
>>
>>
>>
>> rm d
>>
>> rm: cannot remove `d': No space left on device
>>
>>
>>
>> and mds_cache:
>>
>> {
>>
>> "mds_cache": {
>>
>> "num_strays": 999713,
>>
>> "num_strays_purging": 0,
>>
>> "num_strays_delayed": 0,
>>
>> "num_purge_ops": 0,
>>
>> "strays_created": 999723,
>>
>> "strays_purged": 10,
>>
>> "strays_reintegrated": 0,
>>
>> "strays_migrated": 0,
>>
>> "num_recovering_processing": 0,
>>
>> "num_recovering_enqueued": 0,
>>
>> "num_recovering_prioritized": 0,
>>
>> "recovery_started": 107,
>>
>> "recovery_completed": 107
>>
>> }
>>
>> }
>>
>>
>>
>> It seems starys num are stuck, what should I do?
>>
>> Thanks all.
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flist
>> s.ceph.com%2Flistinfo.cgi%2Fceph-users-ceph.com=02%7C01%7C%7C571
>> 893beb17d4d643a8e08d54c26fbed%7C84df9e7fe9f640afb435%7C1%
>> 7C0%7C636498652669125646=%2BUv%2BbpXBv%2B4INeEyjNLML1e%2BEjPMVH
>> k%2FD5iU8a7a1DA%3D=0
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
