Hi Igor,
> Am 24.04.2020 um 13:09 schrieb Igor Fedotov <ifedo...@suse.de>:
>
> that's not 100% pure experiment. Fresh OSD might be faster by itself. E.g.
> due to lack of space fragmentation and/or empty lookup tables.
Also the migrated ones were just 3 weeks old having a usage of 5%.
> You might want to recreate OSD.0 without DB and attach DB manually. Then
> benchmark resulting OSD.
I’ve 24 osds where I migrated db and 8 which where initially created with dB.
All of them show the same symptoms.
> Different experiment if you have another slow OSD with recently added DB
> would be to:
Yes see above 24 of them ;-)
> Compare benchmark results for both bitmap and stupid allocators for this
> specific OSD. I.e. benchmark it as-is then change
> bluestore_allocator/bluefs_allocator to stupid and benchmark again.
Can try in a few hours.
> And just in case - I presume all the benchmark results are persistent, i.e.
> you can see the same results for multiple runs.
Yes I did 10 runs for each posted benchmark.
Thanks,
Stefan
>
>
> Thanks,
>
> Igor
>
>
>
>> On 4/24/2020 12:32 PM, Stefan Priebe - Profihost AG wrote:
>> Hi Igor,
>>
>> there must be a difference. I purged osd.0 and recreated it.
>>
>> Now it gives:
>> ceph tell osd.0 bench
>> {
>> "bytes_written": 1073741824,
>> "blocksize": 4194304,
>> "elapsed_sec": 8.1554735639999993,
>> "bytes_per_sec": 131659040.46819863,
>> "iops": 31.389961354303033
>> }
>>
>> What's wrong wiht adding a block.db device later?
>>
>> Stefan
>>
>>> Am 23.04.20 um 20:34 schrieb Stefan Priebe - Profihost AG:
>>> Hi,
>>>
>>> if the OSDs are idle the difference is even more worse:
>>>
>>> # ceph tell osd.0 bench
>>> {
>>> "bytes_written": 1073741824,
>>> "blocksize": 4194304,
>>> "elapsed_sec": 15.396707875000001,
>>> "bytes_per_sec": 69738403.346825853,
>>> "iops": 16.626931034761871
>>> }
>>>
>>> # ceph tell osd.38 bench
>>> {
>>> "bytes_written": 1073741824,
>>> "blocksize": 4194304,
>>> "elapsed_sec": 6.8903985170000004,
>>> "bytes_per_sec": 155831599.77624846,
>>> "iops": 37.153148597776521
>>> }
>>>
>>> Stefan
>>>
>>> Am 23.04.20 um 14:39 schrieb Stefan Priebe - Profihost AG:
>>>> Hi,
>>>> Am 23.04.20 um 14:06 schrieb Igor Fedotov:
>>>>> I don't recall any additional tuning to be applied to new DB volume. And
>>>>> assume the hardware is pretty the same...
>>>>>
>>>>> Do you still have any significant amount of data spilled over for these
>>>>> updated OSDs? If not I don't have any valid explanation for the phenomena.
>>>>
>>>> just the 64k from here:
>>>> https://tracker.ceph.com/issues/44509
>>>>
>>>>> You might want to try "ceph osd bench" to compare OSDs under pretty the
>>>>> same load. Any difference observed
>>>>
>>>> Servers are the same HW. OSD Bench is:
>>>> # ceph tell osd.0 bench
>>>> {
>>>> "bytes_written": 1073741824,
>>>> "blocksize": 4194304,
>>>> "elapsed_sec": 16.091414781000001,
>>>> "bytes_per_sec": 66727620.822242722,
>>>> "iops": 15.909104543266945
>>>> }
>>>>
>>>> # ceph tell osd.36 bench
>>>> {
>>>> "bytes_written": 1073741824,
>>>> "blocksize": 4194304,
>>>> "elapsed_sec": 10.023828538,
>>>> "bytes_per_sec": 107118933.6419194,
>>>> "iops": 25.539143953780986
>>>> }
>>>>
>>>>
>>>> OSD 0 is a Toshiba MG07SCA12TA SAS 12G
>>>> OSD 36 is a Seagate ST12000NM0008-2H SATA 6G
>>>>
>>>> SSDs are all the same like the rest of the HW. But both drives should give
>>>> the same performance from their specs. The only other difference is that
>>>> OSD 36 was directly created with the block.db device (Nautilus 14.2.7) and
>>>> OSD 0 (14.2.8) does not.
>>>>
>>>> Stefan
>>>>
>>>>>
>>>>> On 4/23/2020 8:35 AM, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> is there anything else needed beside running:
>>>>>> ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD}
>>>>>> bluefs-bdev-new-db --dev-target /dev/vgroup/lvdb-1
>>>>>>
>>>>>> I did so some weeks ago and currently i'm seeing that all osds
>>>>>> originally deployed with --block-db show 10-20% I/O waits while all
>>>>>> those got converted using ceph-bluestore-tool show 80-100% I/O waits.
>>>>>>
>>>>>> Also is there some tuning available to use more of the SSD? The SSD
>>>>>> (block-db) is only saturated at 0-2%.
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
