Hi Igor,
Just to update you on our progress.
- We've not had another repro of this since switching to bitmap allocator /
upgrading to the latest octopus release. I'll try to gather the full set of
diags if we do see this again.
- I think my issues with an empty /var/lib/ceph/osd/ceph-N/ folder are because
we're running ceph in container which is using a mounted filesystem. As soon as
I stop the OSD, the container terminated and the filesystem disappears. There's
probably a way to decouple the ceph process from the lifetime of the container,
but I've not figured it out yet.
Cheers again for all your help,
Dave
-----Original Message-----
From: Igor Fedotov <ifedo...@suse.de>
Sent: 26 July 2021 13:30
To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
Subject: Re: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc loaded
132 GiB in 2930776 extents available 113 GiB"
Dave,
please see inline
On 7/26/2021 1:57 PM, Dave Piper wrote:
> Hi Igor,
>
>> So to get more verbose but less log one can set both debug-bluestore and
>> debug-bluefs to 1/20. ...
> More verbose logging attached. I've trimmed the file to a single restart
> attempt to keep the filesize down; let me know if there's not enough here.
Jul 26 10:25:07 condor_sc0 docker[19100]: -9628>
2021-07-26T10:25:05.512+0000 7f9b3ed48f40 20 bluefs _read got 32768 Jul 26
10:25:07 condor_sc0 docker[19100]: -9627>
2021-07-26T10:25:05.512+0000 7f9b3ed48f40 10 bluefs _read h
0x563e8bd3ff80 0xb2d0000~8000 from file(ino 316842 size 0xe6a476e mtime
2021-07-14T15:54:21.751044+0000 allocated e6b0000 extents
[1:0x1128740000~10000,1:0x1128780000~10000,1:0x112ad60000~10000,1:0x1165fa0000~10000,1:0x11662a0000~10000,1:0x1167410000~10000,1:0x116be10000~10000,1:0x112a330000~20000,1:0x1170890000~10000,1:0x11751e0000~10000,1:0x11770c0000~10000,1:0x1177760000~10000,1:0x1178050000~10000,1:0x117b2d0000~10000,1:0x117b580000~10000,1:0x12a9770000~10000,1:0x12a9820000~10000,1:0x12a98d0000~10000,1:0x12a98f0000~10000,1:0x12a9930000~10000,1:0x12a9a00000~10000,1:0x12a9b30000~10000,1:0x12a9bd0000~10000,1:0x12a9c30000~10000,1:0x12a9c70000~10000,1:0x12a9ca0000~10000,1:0x12a9cc0000~10000,1:0x12a9df0000~10000,1:0x12a9e90000~10000,1:0x12a9ec0000~10000,1:0x12a9f30000~10000,1:0x12a9f90000~10000,1:0x12a9ff0000~10000,1:0x12aa020000~10000,1:0x12aa050000~10000,1:0x12aa090000~10000,1:0x12aa0b0000~10000,1:0x12aa150000~10000,1:0x12aa220000~10000,1:0x12aa370000~10000,1:0x12aa410000~10000,1:0x12aa440000~10000,1:0x12aa490000~10000,1:0x12aa4c0000~10000,1:0x12aa510000~10000,1:0x12aa5a0000~10000,1:0x12aa7b0000~10000,1:0x12aa910000~10000,1:0x12aa950000~10000,1:0x12aa970000~10000,1:0x12aa9a0000~10000,1:0x12aa9d0000~10000,1:0x12aab60000~10000,1:0x12aac10000~10000,1:0x12ad420000~10000,1:0x12ad4d0000~10000,1:0x12ad7a0000~10000,1:0x12ad810000~10000,1:0x12e4450000~10000,1:0x12e7fe0000~10000,1:0x12ecc40000~10000,1:0x12f2510000~10000,1:0x12f2bd0000~10000,1:0x12f3430000~10000,1:0x12f5210000~10000,1:0x132a050000~10000,1:0x133f8e0000~10000,1:0x13433d0000~10000,1:0x1344ef0000~10000,1:0x13478d0000~10000,1:0x134b7f0000~10000,1:0x134d450000~10000,1:0x134dc60000~10000,1:0x134dc80000~10000,1:0x13511c0000~10000,1:0x1351230000~10000,1:0x1352af0000~10000,1:0x1355cc0000~10000,1:0x135ec80000~10000,1:0x136cfb0000~10000,1:0x13737a0000~10000,1:0x1374750000~10000,1:0x137fe70000~10000,1:0x1392590000~10000,1:0x1393a20000~10000,1:0x1396670000~10000,1:0x139d560000~10000,1:0x13a17d0000~10000,1:0x13a20e0000~10000,1:0x13a2ea0000~10000,1:0x13a5420000~10000,1:0x13a6d70000~10000,1:0x13abef0000~10000,1:0x13bc840000~10000,1:0x13caa00000~10000,1:0x13cedf0000~10000,1:0x2a00540000~10000,1:0x2a02630000~10000,1:0x2a04100000~10000,1:0x2a08550000~10000,1:0x2a091e0000~10000,1:0x2a09410000~10000,1:0x2a17c70000~10000,1:0x2a1a860000~10000,1:0x2a1a930000~10000,1:0x2a1fa00000~10000,1:0x2a21bc0000~10000,1:0x2a24d20000~10000,1:0x2a2b5a0000~10000,1:0x2a2c9b0000~10000,1:0x2a36c30000~10000,1:0x2a37c40000~10000,1:0x2a39c60000~10000,1:0x2a43480000~10000,1:0x2a43b80000~10000,1:0x2a46250000~10000,1:0x2a4ba60000~10000,1:0x2a526b0000~10000,1:0x2a55b00000~10000,1:0x2a59920000~10000,1:0x2a5fee0000~10000,1:0x2a686c0000~10000,1:0x2a76280000~10000,1:0x2a8b7d0000~10000,1:0x2a91f00000~10000,1:0x2a98930000~10000,1:0x2a9b4f0000~10000,1:0x2a9e720000~10000,1:0x2aa5920000~10000,1:0x2aac110000~10000,1:0x2ab57f0000~10000,1:0x2ac2690000~10000,1:0x2ac57e0000~10000,1:0x2ad6710000~10000,1:0x2ad9040000~10000,1:0x2afa900000~10000,1:0x2afb110000~10000,1:0x2afd070000~10000,1:0x2b15220000~10000,1:0x2b189b0000~10000,1:0x2b330f0000~10000,1:0x2b33810000~10000,1:0x2b365a0000~10000,1:0x2b36aa0000~10000,1:0x2b36d60000~10000,1:0x2b397b0000~10000,1:0x2b3f510000~10000,1:0x2b42960000~10000,1:0x2b43e60000~10000,1:0x2b5e900000~10000,1:0x2b62060000~10000,1:0x2b63060000~10000,1:0x2b652f0000~10000,1:0x2b7bf60000~10000,1:0x2b876d0000~10000,1:0x2b8a3f0000~10000,1:0x2b94810000~10000,1:0x2b951e0000~10000,1:0x2b957e0000~10000,1:0x2b98a80000~10000,1:0x2b99480000~10000,1:0x2b9b880000~10000,1:0x2ba1030000~10000,1:0x2ba2170000~10000,1:0x2bb6670000~10000,1:0x2bd2870000~10000,1:0x2bd2a80000~10000,1:0x2bed7a0000~10000,1:0x2bf3800000~10000,1:0x2bf44f0000~10000,1:0x2bf8d40000~10000,1:0x2c129b0000~10000,1:0x2c142a0000~10000,1:0x2c171b0000~10000,1:0x2c1abd0000~10000,1:0x2c1d810000~10000,1:0x2c23ea0000~10000,1:0x2c259b0000~10000,1:0x2c25f20000~10000,1:0x2c28330000~10000,1:0x2c320f0000~10000,1:0x2c48430000~10000,1:0x2c56500000~10000,1:0x2c60fa0000~10000,1:0x2c65a40000~10000,1:0x2c68f40000~10000,1:0x2c9c130000~10000,1:0x2ca1350000~10000,1:0x2ca5880000~10000,1:0x2cd4380000~10000,1:0x2ce2b40000~10000,1:0x2cf0e00000~10000,1:0x2d005a0000~10000,1:0x2d0f700000~10000,1:0x2d0fc00000~10000,1:0x2d19ed0000~10000,1:0x2d3ddc0000~10000,1:0x2d3fbc0000~10000,1:0x2d43040000~10000,1:0x2d6de50000~10000,1:0x13bcd60000~20000,1:0x13bf180000~20000,1:0x13c3ec0000~20000,1:0x13c5a00000~20000,1:0x13c6070000~20000,1:0x13ce790000~20000,1:0x13cfc70000~20000,1:0x13d0970000~20000,1:0x2a08d60000~20000,1:0x2a09170000~20000,1:0x2a09b30000~20000,1:0x2a11950000~20000,1:0x2a17890000~20000,1:0x2a28550000~20000,1:0x2a34d20000~20000,1:0x2a57410000~20000,1:0x2a58c00000~20000,1:0x2a5c7c0000~20000,1:0x2a5cf40000~20000,1:0x2a663b0000~20000,1:0x2a68660000~20000,1:0x2a687e0000~20000,1:0x2a73d50000~20000,1:0x2a9fca0000~20000,1:0x2aa5300000~20000,1:0x2aaaaf0000~20000,1:0x2ab7700000~20000,1:0x2ac1d70000~20000,1:0x2ac6f30000~20000,1:0x2ad93a0000~20000,1:0x2ad9ad0000~20000,1:0x2ada2f0000~20000,1:0x2ae5ca0000~20000,1:0x2ae82c0000~20000,1:0x2aebe10000~20000,1:0x13bcd20000~20000,1:0x2b0ff10000~20000,1:0x2b11530000~20000,1:0x2b18790000~20000,1:0x2b35c80000~20000,1:0x2b4d290000~20000,1:0x2b54d70000~20000,1:0x2b5cf40000~20000,1:0x2b5d990000~20000,1:0x2b5e3e0000~20000,1:0x2b5ee10000~20000,1:0x2b5f860000~20000,1:0x2b5fd80000~20000,1:0x2b8fc60000~20000,1:0x2b91fd0000~20000,1:0x2b934f0000~20000,1:0x2ba72b0000~20000,1:0x2baaef0000~20000,1:0x2bb7b80000~20000,1:0x2bbe290000~20000,1:0x2bc7460000~20000,1:0x2bf7200000~20000,1:0x2c09340000~20000,1:0x2c217e0000~20000,1:0x2c231e0000~20000,1:0x2c45f30000~20000,1:0x2c4c4f0000~20000,1:0x2c4c810000~20000,1:0x2c4efa0000~20000,1:0x2c95750000~20000,1:0x2c99b30000~20000,1:0x2c9b130000~20000,1:0x2ca5360000~20000,1:0x2cb9540000~20000,1:0x2cb9c40000~20000,1:0x2cd8e90000~20000,1:0x2d399b0000~20000,1:0x2b05b20000~20000,1:0x2d3d140000~20000,1:0x2d3dca0000~20000,1:0x11754f0000~30000,1:0x12e67e0000~30000,1:0x1304830000~30000,1:0x139eba0000~30000,1:0x13a32e0000~30000,1:0x13b1680000~30000,1:0x13b3c00000~30000,1:0x13b95a0000~30000,1:0x13bd6d0000~30000,1:0x13c3790000~30000,1:0x2a0bea0000~30000,1:0x2a193b0000~30000,1:0x2a19830000~30000,1:0x2a1ab80000~30000,1:0x2a1b570000~30000,1:0x2a29740000~30000,1:0x2a2a170000~30000,1:0x2a2bc00000~30000,1:0x2a2ed80000~30000,1:0x2a36800000~30000,1:0x2a38750000~30000,1:0x2a3e360000~10000,1:0x2d3a330000~20000,1:0x2a43150000~30000,1:0x2a456f0000~30000,1:0x2a54c40000~30000,1:0x2a63f60000~30000,1:0x2a665d0000~30000,1:0x2a6b350000~30000,1:0x2a791a0000~30000,1:0x2a7a5d0000~30000,1:0x2a7f2e0000~30000,1:0x2a846c0000~30000,1:0x2a84df0000~30000,1:0x2aa91f0000~30000,1:0x2aaf450000~30000,1:0x2ab3130000~30000,1:0x2ab4ac0000~30000,1:0x2aca290000~30000,1:0x2addc80000~30000,1:0x2ae9de0000~30000,1:0x2af3770000~30000,1:0x2b04740000~30000,1:0x2b06190000~30000,1:0x2b07620000~30000,1:0x2b09880000~30000,1:0x2b294d0000~10000,1:0x2a3e340000~20000,1:0x2b38870000~30000,1:0x2b5d460000~30000,1:0x2b5deb0000~30000,1:0x2b5f330000~30000,1:0x2b602a0000~30000,1:0x2b64580000~30000,1:0x2b81730000~30000,1:0x2b8b710000~30000,1:0x2b9b990000~30000,1:0x2b9ed00000~30000,1:0x2ba1540000~30000,1:0x2ba5080000~30000,1:0x2bafe50000~30000,1:0x2bd3ab0000~10000,1:0x2b294e0000~20000,1:0x2bd5890000~30000,1:0x2bdc5d0000~30000,1:0x2bdc900000~30000,1:0x2bf19b0000~30000,1:0x2c08e10000~30000,1:0x2c25bc0000~30000,1:0x2c33c00000~30000,1:0x2c509d0000~30000,1:0x2c67770000~30000,1:0x2c6f5a0000~30000,1:0x2c784e0000~30000,1:0x2c8c030000~10000,1:0x2bd3a90000~20000,1:0x2c8f2e0000~30000,1:0x2c97b30000~30000,1:0x2cbb770000~30000,1:0x2cc0290000~30000,1:0x2cc7110000~30000,1:0x2cd3ef0000~30000,1:0x2cde190000~30000,1:0x2cf6f10000~30000,1:0x2d3ce10000~30000,1:0x2d63960000~30000,1:0x1125ef0000~40000,1:0x112ac00000~40000,1:0x112ac80000~40000,1:0x1162280000~40000,1:0x11686e0000~20000,1:0x2c8c010000~20000,1:0x116adf0000~40000,1:0x116dd40000~40000,1:0x1170380000~40000,1:0x1175ac0000~40000,1:0x1176e80000~40000,1:0x11777c0000~40000,1:0x1177820000~40000,1:0x1178270000~40000,1:0x11787f0000~40000,1:0x11686c0000~20000,1:0x117fc30000~20000,1:0x11807e0000~40000,1:0x1183a90000~40000,1:0x11845d0000~40000,1:0x118aad0000~40000,1:0x118d230000~40000,1:0x118d4a0000~40000,1:0x1196d80000~40000,1:0x12a84c0000~40000,1:0x12ae7c0000~20000,1:0x117fc10000~20000,1:0x12b01a0000~40000,1:0x12b2710000~40000,1:0x12c70b0000~40000,1:0x12cd280000~40000,1:0x12cee60000~40000,1:0x12d4690000~40000,1:0x12d8a90000~40000,1:0x12dd5e0000~40000,1:0x12df360000~40000,1:0x12e5e70000~10000,1:0x12e5ea0000~10000,1:0x12ae7e0000~20000,1:0x12e6b10000~40000,1:0x12e9730000~40000,1:0x12e9870000~40000,1:0x12ea390000~40000,1:0x12eae80000~40000,1:0x12ec6b0000~40000,1:0x1312b90000~40000,1:0x13217e0000~40000,1:0x1327060000~40000,1:0x12e5e80000~20000,1:0x1333780000~20000,1:0x1338b00000~40000,1:0x1338ef0000~40000,1:0x1339290000~40000,1:0x133e320000~40000,1:0x133ebe0000~40000,1:0x1341650000~40000,1:0x13467e0000~40000,1:0x13469e0000~40000,1:0x1333760000~20000,1:0x1347f10000~20000,1:0x1348c70000~40000,1:0x134c250000~40000,1:0x13502b0000~40000,1:0x13685c0000~40000,1:0x136b9f0000~40000,1:0x136cb60000~40000,1:0x136ce00000~40000,1:0x13731d0000~40000,1:0x13796c0000~20000,1:0x1347ef0000~20000,1:0x137b700000~40000,1:0x1384a40000~40000,1:0x138ee10000~40000,1:0x1399730000~40000,1:0x13b8130000~40000,1:0x13bb250000~40000,1:0x13c3ad0000~40000,1:0x13c5c20000~40000,1:0x13c7320000~10000,1:0x13c7350000~10000,1:0x13796e0000~20000,1:0x13c74e0000~40000,1:0x13c8180000~40000,1:0x13cb700000~40000,1:0x13cbc40000~40000,1:0x2a06450000~40000,1:0x2a0d440000~40000,1:0x2a17450000~40000,1:0x2a1c4d0000~40000,1:0x2a300b0000~10000,1:0x2a300e0000~10000,1:0x13c7330000~20000,1:0x2a37f50000~40000,1:0x2a39320000~40000,1:0x2a3e690000~40000,1:0x2a47d50000~40000,1:0x2a4d5b0000~40000,1:0x2a555d0000~40000,1:0x2a77a60000~40000,1:0x2a7de60000~40000,1:0x2a300c0000~20000,1:0x2a888a0000~20000,1:0x2a8a670000~40000,1:0x2a97750000~40000,1:0x2a9cec0000~40000,1:0x2aa8970000~40000,1:0x2aa8bb0000~40000,1:0x2ac61e0000~40000,1:0x2ad5cc0000~40000,1:0x2adc150000~40000,1:0x2a88880000~20000,1:0x2afd290000~20000,1:0x2b06880000~40000,1:0x2b09db0000~40000,1:0x2b16790000~40000,1:0x2b26a90000~40000,1:0x2b2bf90000~40000,1:0x2b2f9d0000~40000,1:0x2b48570000~40000,1:0x2b4f890000~40000,1:0x2b5ca00000~10000,1:0x2b5ca30000~10000,1:0x2afd270000~20000,1:0x2b604d0000~40000,1:0x2b61620000~40000,1:0x2b69e50000~40000,1:0x2b6fb00000~40000,1:0x2b7a820000~40000,1:0x2b91080000~40000,1:0x2b964f0000~40000,1:0x2b97f70000~40000,1:0x2b98f80000~20000,1:0x2b5ca10000~20000,1:0x2bb3590000~40000,1:0x2bcdca0000~40000,1:0x2bda690000~40000,1:0x2be93c0000~40000,1:0x2bfd6b0000~40000,1:0x2c35b40000~40000,1:0x2c49df0000~40000,1:0x2c5f560000~40000,1:0x2c9d240000~20000,1:0x2b98fa0000~20000,1:0x2cc4a50000~40000,1:0x2ccb1c0000~40000,1:0x2cda090000~40000,1:0x2ced0b0000~40000,1:0x2d3d260000~40000,1:0x2d43a40000~40000,1:0x2d7f160000~40000,1:0x2d7fdb0000~40000,1:0x11259b0000~50000,1:0x112a9b0000~30000,1:0x2c9d260000~20000,1:0x112aab0000~50000,1:0x116a9a0000~50000,1:0x1174ed0000~50000,1:0x1178990000~50000,1:0x117f3a0000~50000,1:0x1183c60000~50000,1:0x118a020000~50000,1:0x118b320000~50000,1:0x112a9e0000~20000,1:0x118c9f0000~30000,1:0x12964f0000~50000,1:0x12acdd0000~50000,1:0x12b16b0000~50000,1:0x12d8cd0000~50000,1:0x12e78d0000~50000,1:0x12ee0c0000~50000,1:0x12ff530000~50000,1:0x12ff780000~10000,1:0x118c9d0000~20000,1:0x12ff7b0000~20000,1:0x1306460000~50000,1:0x13073b0000~50000,1:0x130d8e0000~50000,1:0x1315b40000~50000,1:0x131da80000~50000,1:0x1324a10000~50000,1:0x1325360000~20000,1:0x13253a0000~10000,1:0x12ff790000~20000,1:0x1331830000~50000,1:0x133d7f0000~50000,1:0x1343ca0000~50000,1:0x1350a80000~50000,1:0x13570a0000~50000,1:0x1358080000~50000,1:0x135ae50000~50000,1:0x135bb30000~20000,1:0x135bb70000~10000,1:0x1325380000~20000,1:0x1360ca0000~50000,1:0x1368510000~50000,1:0x1368f00000~50000,1:0x136b5a0000~50000,1:0x1377320000~50000,1:0x1377470000~50000,1:0x1377640000~50000,1:0x135bb50000~20000,1:0x137b010000~30000,1:0x13896b0000~50000,1:0x13981a0000~50000,1:0x13a0980000~50000,1:0x13bc9d0000~50000,1:0x13be130000~50000,1:0x13c0be0000~50000,1:0x137aff0000~20000,1:0x13c3160000~30000,1:0x13c3c10000~50000,1:0x13c49a0000~50000,1:0x13c6280000~50000,1:0x13c6f90000~50000,1:0x13cc5f0000~50000,1:0x13cf5e0000~50000,1:0x2a038b0000~50000,1:0x13c3140000~20000,1:0x2a065c0000~30000,1:0x2a140b0000~50000,1:0x2a14e00000~50000,1:0x2a17d80000~50000,1:0x2a1ec10000~50000,1:0x2a318d0000~50000,1:0x2a38ae0000~50000,1:0x2a38d50000~30000,1:0x2a065a0000~20000,1:0x2a40ab0000~50000,1:0x2a43dd0000~50000,1:0x2a4c060000~50000,1:0x2a4ee30000~50000,1:0x2a597d0000~50000,1:0x2a655f0000~50000,1:0x2a67700000~50000,1:0x2a38d80000~20000,1:0x2a7c9f0000~30000,1:0x2a84070000~50000,1:0x2ab4760000~50000,1:0x2abed50000~50000,1:0x2ac7d50000~50000,1:0x2ac8560000~50000,1:0x2acaf20000~50000,1:0x2ad42f0000~20000,1:0x2ad4330000~10000,1:0x2a7c9d0000~20000,1:0x2af58b0000~50000,1:0x2afb420000~50000,1:0x2affa60000~50000,1:0x2b03920000~50000,1:0x2b05d40000~50000,1:0x2b06fc0000~50000,1:0x2b12ce0000~50000,1:0x2b17860000~20000,1:0x2b178a0000~10000,1:0x2ad4310000~20000,1:0x2b4fd10000~50000,1:0x2b97520000~50000,1:0x2b9e3b0000~50000,1:0x2be5ec0000~50000,1:0x2c11280000~50000,1:0x2c29b60000~50000,1:0x2c2a7b0000~50000,1:0x2c2de50000~30000,1:0x2b17880000~20000,1:0x2c6ba50000~50000,1:0x2c9d680000~50000,1:0x11622d0000~60000,1:0x1166c20000~60000,1:0x1171050000~60000,1:0x1172020000~60000,1:0x11771d0000~30000,1:0x1177220000~10000,1:0x2c2de80000~20000,1:0x1177870000~60000,1:0x117abd0000~60000,1:0x117b840000~60000,1:0x117ff90000~60000,1:0x1183a20000~60000,1:0x118b110000~40000,1:0x1177200000~20000,1:0x118b180000~60000,1:0x118e1d0000~60000,1:0x128cdf0000~60000,1:0x128ce60000~60000,1:0x12a82a0000~10000,1:0x118b150000~20000,1:0x12a82d0000~30000,1:0x12b39e0000~60000,1:0x12c03a0000~60000,1:0x12c9310000~60000,1:0x12cbc50000~60000,1:0x12a82b0000~20000,1:0x12da380000~40000,1:0x12e7490000~60000,1:0x12ec7c0000~60000,1:0x12ede10000~60000,1:0x12ee660000~40000,1:0x12da360000~20000,1:0x12ef4c0000~60000,1:0x12f1490000~60000,1:0x130e480000~60000,1:0x130f3a0000~60000,1:0x1311430000~10000,1:0x12ee6a0000~20000,1:0x1311460000~30000,1:0x1311a90000~60000,1:0x1311d00000~60000,1:0x1312260000~60000,1:0x131e4f0000~40000,1:0x1311440000~20000,1:0x131f360000~60000,1:0x1322ab0000~60000,1:0x1347d50000~60000,1:0x134ef60000~60000,1:0x135b0f0000~20000,1:0x131e530000~20000,1:0x135b130000~20000,1:0x1369950000~60000,1:0x136a480000~60000,1:0x137e590000~60000,1:0x137eb30000~40000,1:0x135b110000~20000,1:0x1382d30000~60000,1:0x138e070000~60000,1:0x1395e40000~60000,1:0x1397330000~60000,1:0x13a0cd0000~10000,1:0x137eb70000~20000,1:0x13a0d00000~30000,1:0x13c9ff0000~60000,1:0x13cc9f0000~60000,1:0x2a08090000~60000,1:0x2a252c0000~30000,1:0x2a25310000~10000,1:0x13a0ce0000~20000,1:0x2a33bc0000~60000,1:0x2a38250000~60000,1:0x2a41100000~60000,1:0x2a436e0000~60000,1:0x2a252f0000~20000,1:0x2a79bb0000~40000,1:0x2a80a70000~60000,1:0x2a8aac0000~60000,1:0x2a9d9c0000~60000,1:0x2ab09f0000~30000,1:0x2ab0a40000~10000,1:0x2a79b90000~20000,1:0x2abd150000~60000,1:0x2ac0950000~60000,1:0x2ac10f0000~60000,1:0x2ada7e0000~60000,1:0x2ab0a20000~20000,1:0x2af8660000~40000,1:0x2b02800000~60000,1:0x2b02960000~60000,1:0x2b031c0000~60000,1:0x2af8640000~20000,1:0x2b15350000~40000,1:0x2b344c0000~60000,1:0x2b35520000~60000,1:0x2b3a5d0000~60000,1:0x2b97710000~20000,1:0x2b15330000~20000,1:0x2b97750000~20000,1:0x2bc22f0000~60000,1:0x2bdf940000~60000,1:0x2bea590000~60000,1:0x2bf59f0000~40000,1:0x2b97730000~20000,1:0x2bf9450000~60000,1:0x2c00080000~60000,1:0x2c75870000~60000,1:0x2c93de0000~60000,1:0x2bf5a30000~20000,1:0x2c9dc00000~40000,1:0x2cc47f0000~60000,1:0x2cd4220000~60000,1:0x2cec450000~60000,1:0x2d3a0d0000~20000,1:0x2d3a110000~20000,1:0x2c9dbe0000~20000,1:0x2d3c940000~60000,1:0x11663b0000~70000,1:0x1169f00000~70000,1:0x116a370000~30000,1:0x116a3c0000~20000,1:0x2d3a0f0000~20000,1:0x116d510000~70000,1:0x116f290000~70000,1:0x117f2e0000~70000,1:0x1182780000~20000,1:0x116a3a0000~20000,1:0x11827c0000~30000,1:0x1188270000~70000,1:0x1189c90000~70000,1:0x118ac20000~70000,1:0x118d0b0000~20000,1:0x11827a0000~20000,1:0x118d0f0000~30000,1:0x118d6c0000~70000,1:0x118e440000~70000,1:0x128ccd0000~70000,1:0x128ced0000~20000,1:0x118d0d0000~20000,1:0x128cf10000~30000,1:0x128dbb0000~70000,1:0x1297320000~70000,1:0x12a5500000~70000,1:0x12b9b40000~20000,1:0x128cef0000~20000,1:0x12b9b80000~30000,1:0x12e03b0000~70000,1:0x12e5d00000~70000,1:0x12e6670000~70000,1:0x12f29c0000~10000,1:0x12b9b60000~20000,1:0x12f29f0000~40000,1:0x1300220000~70000,1:0x1300480000~70000,1:0x131c0c0000~70000,1:0x131c230000~10000,1:0x12f29d0000~20000,1:0x131c260000~40000,1:0x1323350000~70000,1:0x132afe0000~70000,1:0x132c340000~70000,1:0x131c240000~20000,1:0x1331ba0000~50000,1:0x1331df0000~70000,1:0x1335830000~70000,1:0x1336e50000~50000,1:0x1331b80000~20000,1:0x1340220000~70000,1:0x134da70000~70000,1:0x1350580000~70000,1:0x1353cb0000~40000,1:0x1353d10000~10000,1:0x1336ea0000~20000,1:0x1360630000~70000,1:0x13648d0000~70000,1:0x1369190000~70000,1:0x1371770000~50000,1:0x1353cf0000~20000,1:0x1373c10000~70000,1:0x137fdf0000~70000,1:0x1385a80000~70000,1:0x138a460000~30000,1:0x13717c0000~20000,1:0x138a4b0000~20000,1:0x138ad60000~70000,1:0x138ef70000~70000,1:0x13986f0000~70000,1:0x139e190000~70000,1:0x13afbc0000~40000,1:0x13afc20000~10000,1:0x138a490000~20000,1:0x13bc490000~70000,1:0x13bd080000~70000,1:0x13c3e10000~70000,1:0x13cb8d0000~70000,1:0x13cc340000~70000,1:0x13afc00000~20000,1:0x29ffe30000~50000,1:0x2a09670000~70000,1:0x2a10b00000~70000,1:0x2a20210000~70000,1:0x2a32650000~70000,1:0x2a3a230000~30000,1:0x29ffe10000~20000,1:0x2a3a280000~20000,1:0x2a4bdf0000~70000,1:0x2a54e90000~70000,1:0x2a56aa0000~70000,1:0x2a79cf0000~70000,1:0x2a7ac00000~50000,1:0x2a3a260000~20000,1:0x2a7c060000~70000,1:0x2a85e10000~70000,1:0x2a93f70000~70000,1:0x2aa12c0000~70000,1:0x2ac3160000~70000,1:0x2a7ac50000~20000,1:0x2ad15d0000~50000,1:0x2b018b0000~70000,1:0x2b0bf80000~70000,1:0x2b419f0000~70000,1:0x2b5c490000~70000,1:0x2c04d10000~40000,1:0x2c04d70000~10000,1:0x2ad15b0000~20000,1:0x2c0d910000~70000,1:0x2c0fa80000~70000,1:0x2c150c0000~70000,1:0x2c192b0000~70000,1:0x2c383a0000~70000,1:0x2c4eb30000~10000,1:0x2c04d50000~20000,1:0x2c4eb60000~40000,1:0x2cc7340000~70000,1:0x2ce1dd0000~70000,1:0x2d00bb0000~70000,1:0x2d083c0000~70000,1:0x2d414d0000~50000,1:0x2c4eb40000~20000,1:0x1125570000~80000,1:0x1125de0000~80000,1:0x112acd0000~80000,1:0x116c310000~80000,1:0x11708b0000~30000,1:0x2d41520000~20000,1:0x1170900000~30000,1:0x1172cb0000~80000,1:0x1175720000~80000,1:0x11792f0000~80000,1:0x117baa0000~80000,1:0x117e810000~30000,1:0x11708e0000~20000,1:0x117e860000~30000,1:0x117ef60000~80000,1:0x1183350000~80000,1:0x1185d90000~80000,1:0x1188c80000~80000,1:0x117e840000~20000,1:0x118d760000~60000,1:0x118d890000~80000,1:0x128d8b0000~80000,1:0x12981c0000~80000,1:0x12a7ed0000~60000,1:0x118d740000~20000,1:0x12ad1b0000~80000,1:0x12ad860000~80000,1:0x12cb1a0000~80000,1:0x12cece0000~80000,1:0x12da0e0000~30000,1:0x12a7f30000~20000,1:0x12da130000~30000,1:0x12e6c50000~80000,1:0x12eac00000~80000,1:0x12eb0c0000~80000,1:0x12eb840000~80000,1:0x12f1c90000~20000,1:0x12da110000~20000,1:0x12f1cd0000~40000,1:0x12ffea0000~80000,1:0x130ce30000~80000,1:0x130e6a0000~80000,1:0x130fa60000~80000,1:0x12f1cb0000~20000,1:0x13237a0000~60000,1:0x132ac60000~80000,1:0x132b230000~80000,1:0x132c0c0000~80000,1:0x13387e0000~80000,1:0x1323780000~20000,1:0x133c6a0000~60000,1:0x13414d0000~80000,1:0x13485b0000~80000,1:0x134eaa0000~80000,1:0x1353ab0000~60000,1:0x133c680000~20000,1:0x13560a0000~80000,1:0x1366b40000~80000,1:0x1367960000~80000,1:0x13683d0000~80000,1:0x1370260000~50000,1:0x13702d0000~10000,1:0x1353b10000~20000,1:0x13743d0000~80000,1:0x1383b20000~80000,1:0x138a9b0000~80000,1:0x138c6e0000~80000,1:0x1390ee0000~30000,1:0x13702b0000~20000,1:0x1390f30000~30000,1:0x13928b0000~80000,1:0x1398d80000~80000,1:0x139f970000~80000,1:0x13b1ab0000~80000,1:0x1390f10000~20000,1:0x13ba5f0000~60000,1:0x13ca490000~80000,1:0x13d10f0000~80000,1:0x2a05d50000~80000,1:0x2a075d0000~60000,1:0x13ba5d0000~20000,1:0x2a224d0000~80000,1:0x2a2e600000~80000,1:0x2a390a0000~80000,1:0x2a481f0000~80000,1:0x2a4bc70000~30000,1:0x2a07630000~20000,1:0x2a4bcc0000~30000,1:0x2a5a6c0000~80000,1:0x2a5d160000~80000,1:0x2a75700000~80000,1:0x2a7dce0000~80000,1:0x2a86e90000~20000,1:0x2a4bca0000~20000,1:0x2a86ed0000~40000,1:0x2a8d920000~80000,1:0x2a98f40000~80000,1:0x2abae10000~60000,1:0x2a86eb0000~20000,1:0x2ad2320000~80000,1:0x2b00d30000~80000,1:0x2b07c10000~80000,1:0x2b66200000~40000,1:0x2abae70000~20000,1:0x2b66260000~20000,1:0x2b6ba90000~80000,1:0x2b7d880000~80000,1:0x2be9700000~80000,1:0x2c15230000~10000,1:0x2b66240000~20000,1:0x2c15260000~50000,1:0x2c2f160000~80000,1:0x2cad5a0000~80000,1:0x2ce3050000~50000,1:0x2ce30c0000~10000,1:0x2c15240000~20000,1:0x2ce3cd0000~80000,1:0x2d0f980000~80000,1:0x2d59e90000~80000,1:0x2d5a210000~10000,1:0x2ce30a0000~20000,1:0x2d5a240000~50000,1:0x11279a0000~90000,1:0x1171430000~90000,1:0x1175f00000~30000,1:0x2d5a220000~20000,1:0x1175f50000~40000,1:0x1177b00000~90000,1:0x11781d0000~90000,1:0x11782f0000~70000,1:0x1175f30000~20000,1:0x117b3e0000~90000,1:0x11803d0000~90000,1:0x1180650000~90000,1:0x1178360000~20000,1:0x1187320000~70000,1:0x11884f0000~90000,1:0x118c930000~90000,1:0x118d610000~10000,1:0x1187300000~20000,1:0x118d640000~60000,1:0x118d7f0000~90000,1:0x1284d20000~90000,1:0x128cd50000~20000,1:0x118d620000~20000,1:0x128cd90000~50000,1:0x1297030000~90000,1:0x12aec10000~90000,1:0x12b2f50000~40000,1:0x128cd70000~20000,1:0x12b2fb0000~30000,1:0x12b4530000~90000,1:0x12ba3b0000~90000,1:0x12d58e0000~60000,1:0x12d5960000~10000,1:0x12b2f90000~20000,1:0x12dd820000~80000])
Jul 26 10:25:07 condor_sc0 docker[19100]: -9626>
2021-07-26T10:25:05.512+0000 7f9b3ed48f40 20 bluefs _read fetching
0x0~80000 of 1:0x1185d90000~80000
Jul 26 10:25:07 condor_sc0 docker[19100]: -9625>
2021-07-26T10:25:05.513+0000 7f9b3ed48f40 20 bluefs _read left 0x80000 len
0x8000
^^^ I expected this stuff to count up to -1 where the most interesting stuff
should be...
Jul 26 10:25:09 condor_sc0 docker[19550]: Error response from daemon: No
such container: ceph-osd-1
Jul 26 10:25:19 condor_sc0 systemd[1]: ceph-osd@1.service holdoff time
over, scheduling restart.
Jul 26 10:25:19 condor_sc0 systemd[1]: Stopped Ceph OSD.
Jul 26 10:25:19 condor_sc0 systemd[1]: Starting Ceph OSD...
>
>> It would be also great to collect the output for the following commands: ...
> I've tried running ceph-bluestore-tool previously on this system but both
> commands fails with the following error:
>
> [qs-admin@condor_sc0 ~]$ sudo docker exec 419d997e5a05 ceph-bluestore-tool
> --path /var/lib/ceph/osd/ceph-1 --command bluefs-stats
> error from cold_open: (11) Resource temporarily unavailable
> 2021-07-26T10:50:15.032+0000 7f9a9bf68240 -1
> bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock
> /var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still running?)(11)
> Resource temporarily unavailable
> [qs-admin@condor_sc0 ~]$
>
> There's only one OSD running on this server; should I be stopping it / the
> other OSDs in the cluster before running the `ceph-bluestore-tool` command?
> Previously when the OSDs were failing to start, /var/lib/ceph/osd/ceph-1/ was
> empty but it now contains the following:
Each folder under /var/lib/ceph/osd/ceph-N/ should be used exclusively
by a single OSD. ceph-bluestore-tool to be run when the respective OSD
is shutdown, other OSDs referring to different folders shouldn't matter.
Is the above valid for your case?
>
> [qs-admin@condor_sc0 ~]$ sudo docker exec 419d997e5a05 ls
> /var/lib/ceph/osd/ceph-1
> block
> ceph_fsid
> fsid
> keyring
> ready
> require_osd_release
> type
> whoami
> [qs-admin@condor_sc0 ~]$
>
>
>> And finally you can try to switch to bitmap allocator as a workaround ...
> Switching to the bitmap allocator as you suggested has led to both failing
> OSDs starting up successfully. I've now got 3/3 OSDs up and in! The cluster
> still has MDS issues that were blocked behind getting the OSDs running as
> mentioned in my original post, but I think these are unrelated to the OSD
> problem as it's an issue we've seen in isolation elsewhere.
>
> So - that's a big step forward! Should I retry with my original config on the
> latest octopus release and see if this is now fixed?
yeah, looks like this is a hybrid allocator bug - hence you can upgrade
and check if this is fixed.
>
> Cheers again,
>
> Dave
>
>
> -----Original Message-----
> From: Igor Fedotov <ifedo...@suse.de>
> Sent: 26 July 2021 11:14
> To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
> Subject: Re: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc
> loaded 132 GiB in 2930776 extents available 113 GiB"
>
> Hi Dave,
>
> Some notes first:
>
> 1) The following behavior is fine, BlueStore mounts in two stages - the first
> one is read-only and among other things it loads allocation map from DB. And
> that's exactly the case here.
>
> Jul 26 08:55:31 condor_sc0 docker[15282]: 2021-07-26T08:55:31.703+0000
> 7f0e15b3df40 1 bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded
> 132 GiB in 2930776 extents available 113 GiB Jul 26 08:55:31 condor_sc0
> docker[15282]: 2021-07-26T08:55:31.703+0000
> 7f0e15b3df40 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all
> background work Jul 26 08:55:31 condor_sc0 docker[15282]:
> 2021-07-26T08:55:31.704+0000
> 7f0e15b3df40 4 rocksdb: [db/db_impl.cc:563] Shutdown complete
>
> 2) What's really broken is the following allocation attempt:
>
> Jul 26 08:55:34 condor_sc0 docker[15282]: 2021-07-26T08:55:34.767+0000
> 7f0e15b3df40 1 bluefs _allocate failed to allocate 0x100716 on bdev 1, free
> 0xd0000; fallback to bdev 2 Jul 26 08:55:34 condor_sc0 docker[15282]:
> 2021-07-26T08:55:34.767+0000
> 7f0e15b3df40 1 bluefs _allocate unable to allocate 0x100716 on bdev 2, free
> 0xffffffffffffffff; fallback to slow device expander Jul 26 08:55:35
> condor_sc0 docker[15282]: 2021-07-26T08:55:35.042+0000
> 7f0e15b3df40 -1 bluestore(/var/lib/ceph/osd/ceph-1)
> allocate_bluefs_freespace failed to allocate on 0x40000000 min_size
> 0x110000 > allocated total 0x0 bluefs_shared_alloc_size 0x10000 allocated 0x0
> available 0x 1c09738000 Jul 26 08:55:35 condor_sc0 docker[15282]:
> 2021-07-26T08:55:35.044+0000
> 7f0e15b3df40 -1 bluefs _allocate failed to expand slow device to fit
> +0x100716
> Jul 26 08:55:35 condor_sc0 docker[15282]: 2021-07-26T08:55:35.044+0000
> 7f0e15b3df40 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
> 0x100716
>
> This occurs during BlueFS recovery and that's an attempt to get more space to
> write out the bluefs log. This shouldn't fail given the plenty of free space:
>
> ... available 0x 1c09738000 ...
>
>
> So to get more verbose but less log one can set both debug-bluestore and
> debug-bluefs to 1/20. This way just last 10000 lines of the log preceeding
> the crash would be at level 20. Which seems sufficient for the
> troubleshooting.
>
> It would be also great to collect the output for the following commands:
>
> ceph-bluestore-tool --path <osd-dir> --command bluefs-bdev-sizes
>
> ceph-bluestore-tool --path <osd-dir> --command bluefs-stats
>
>
> And finally you can try to switch to bitmap allocator as a workaround - we've
> fixed a couple of issues in Hybrid one which prevented from proper
> allcoations under some circumstances. The fixes were made after v15.2.11
> release hence this might be the case. So please try setting:
>
> bluestore_allocator = bitmap
>
> bluefs_allocator = bitmap
>
>
> Thanks,
>
> Igor
>
>
> On 7/26/2021 12:14 PM, Dave Piper wrote:
>> Hi Igor,
>>
>> Thanks for your time looking into this.
>>
>> I've attached a 5 minute window of OSD logs, which includes several restart
>> attempt (each one takes ~25 seconds).
>>
>> When I said it looked like we were starting up in a different state, I'm
>> referring to how "Recovered from manifest file" log appears twice, with
>> different logs afterwards. This behaviour seems to repeat reliably on each
>> restart of the OSD. My interpretation of this was that when the initial
>> recovery attempt leads to the rocksdb shutdown, ceph is automatically trying
>> to start the OSD in some alternative state but that this is also failing
>> (with the bdev errors I copied in). Possibly I'm inferring too much.
>>
>> I tried turning up the logging levels for rocksdb and bluestore but they're
>> both very spammy so I've not included this in the attached logs. Let me know
>> if you think that would be helpful.
>>
>> My ceph version is 15.2.11. We're running a containerized deployment using
>> docker image ceph-daemon:v5.0.10-stable-5.0-octopus-centos-8 .
>>
>> [qs-admin@condor_sc0 metaswitch]$ sudo docker exec b732f9135b42 ceph
>> version ceph version 15.2.11
>> (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)
>>
>> Cheers,
>>
>> Dave
>>
>>
>>
>> -----Original Message-----
>> From: Igor Fedotov <ifedo...@suse.de>
>> Sent: 23 July 2021 20:45
>> To: Dave Piper <david.pi...@microsoft.com>; ceph-users@ceph.io
>> Subject: [EXTERNAL] Re: [ceph-users] OSDs flapping with "_open_alloc loaded
>> 132 GiB in 2930776 extents available 113 GiB"
>>
>> Hi Dave,
>>
>> The follow log line indicates that allocator has just completed loading
>> information about free disk blocks into memory. And it looks perfectly fine.
>>
>>> _open_alloc loaded 132 GiB in 2930776 extents available 113 GiB
>>
>>
>> Subsequent rocksdb shutdown looks weird without any other log output
>> indicating the issue.
>> Curious what do you mean under "
>>
>> After that we seem to try starting up in a slightly different state and get
>> a different set of errors:
>>
>> "
>> The resulted errors show lack of disk space at some point but I'd definitely
>> like to get the full startup log.
>>
>> Please also specify which Octopus version do you have?
>>
>> Thanks,
>> Igor
>>
>> On 7/23/2021 6:48 PM, Dave Piper wrote:
>>> Hi all,
>>>
>>> We've got a containerized test cluster with 3 OSDs and ~ 220GiB of data.
>>> Shortly after upgrading from nautilus -> octopus, 2 of the 3 OSDs have
>>> started flapping. I've also got alarms about the MDS being damaged, which
>>> we've seen elsewhere and have a recovery process for, but I'm unable to run
>>> this (I suspect because I've only got 1 functioning OSD). My RGWs are also
>>> failing to start, again I suspect because of the bad state of OSDs. I've
>>> tried restarting all OSDs, rebooting all servers, checked auth (all looks
>>> fine) - but I'm still in the same state.
>>>
>>> My OSDs seem to be failing at the "_open_alloc opening allocation
>>> metadata" step; looking at logs for each OSD restart, the OSD writes this
>>> log, then no logs for a few minutes and then logs:
>>>
>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB in
>>> 2930776 extents available 113 GiB
>>> rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background
>>> work
>>>
>>> After that we seem to try starting up in a slightly different state and get
>>> a different set of errors:
>>>
>>> bluefs _allocate failed to allocate 0x100716 on bdev 1, free
>>> 0xd0000; fallback to bdev 2
>>> bluefs _allocate unable to allocate 0x100716 on bdev 2, free
>>> 0xffffffffffffffff; fallback to slow device expander
>>>
>>> and eventually crash and log a heap of stack dumps.
>>>
>>> I don't know what extents are but I seem to have a lot of them, and more
>>> than I've got capacity for? Maybe I'm running out of RAM or disk space
>>> somewhere, but I've got 21GB of free RAM on the server, and each OSD has a
>>> 350GiB device attached to it.
>>>
>>>
>>>
>>> I'm wondering if anyone has seen anything like this before or can suggest
>>> next debug steps to take?
>>>
>>> Cheers,
>>>
>>> Dave
>>>
>>>
>>>
>>> Full OSD logs surrounding the "_open_alloc opening allocation metadata"
>>> step:
>>>
>>>
>>> Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1
>>> {"time_micros": 1626998833819439, "job": 1, "event":
>>> "recovery_started", "log_files": [392088, 392132]}
>>>
>>> Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb:
>>> [db/db_impl_open.cc:583] Recovering log #392088 mode 0
>>>
>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:17.240+0000 7f3de111bf40 4 rocksdb:
>>> [db/db_impl_open.cc:583] Recovering log #392132 mode 0
>>>
>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:17.486+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1
>>> {"time_micros": 1626998837486404, "job": 1, "event":
>>> "recovery_finished"}
>>>
>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:17.486+0000 7f3de111bf40 1
>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_db opened rocksdb path db
>>> options
>>> compression=kNoCompression,max_write_buffer_number=4,min_write_buffer
>>> _
>>> number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,
>>> w
>>> ritable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_
>>> b
>>> ackground_compactions=2
>>>
>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist init
>>>
>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist
>>> _init_from_label
>>>
>>> Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:17.529+0000 7f3de111bf40 1
>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc opening allocation
>>> metadata
>>>
>>> Jul 23 00:07:18 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:18.238+0000 7f3de111bf40 1 HybridAllocator
>>> _spillover_range constructing fallback allocator
>>>
>>> Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:20.563+0000 7f3de111bf40 1
>>> bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB in
>>> 2930776 extents available 113 GiB
>>>
>>> Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:20.563+0000 7f3de111bf40 4 rocksdb:
>>> [db/db_impl.cc:390] Shutdown: canceling all background work
>>>
>>> Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]:
>>> 2021-07-23T00:07:20.565+0000 7f3de111bf40 4 rocksdb:
>>> [db/db_impl.cc:563] Shutdown complete
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
>>> email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
