FWIW, I'd be for trying to take periodic samples of actual IO happening
on the drive during operation. You can get a much better idea of
latency and throughput characteristics across different IO sizes over
time (though you will need to account for varying levels of concurrency
at the device
Frank, Sven,
I totally agree with your thoughts on performing a proper benchmark study.
The
osd bench results we obtained when compared with Fio results didn't vary
wildly
and therefore the osd bench was adopted considering the minimal impact to
the
osd startup time. But this apparently is not
Hi all,
My experience with osd bench is not good either. Firstly, its results are
> very fluctuating and secondly it always returns unrealistically large
> numbers. I always get >1000 IOPs for HDDs that can do 100-150 in reality.
> My suspicion is that the internal benchmark is using an IO path
On Do, 2022-09-08 at 08:22 +, Frank Schilder wrote:
> My experience with osd bench is not good either
it seems it was recently "fixed" by writing "a"'s instead of zeroes:
https://github.com/ceph/ceph/commit/db045e005fab218f2bb270b7cb60b62abbbe3619
tongue in cheek:
not sure that this is a
: Sridhar Seshasayee
Sent: Wednesday, September 7, 2022 8:58 AM
To: Vladimir Brik
Cc: David Orman; ceph-users
Subject: [ceph-users] Re: Wide variation in osd_mclock_max_capacity_iops_hdd
Hi Vladimir,
Yes, device caching is disabled. The ODSs in question use a
> separate DB/WAL device on flash.
> I repeated cache drop/bench a few times and the results are
> consistent but high: 1500-1600 IOPS
>
>
Couple of asks:
- Could you please paste the output of the bench command. I wanted to
check how long the test took.
- The Ceph version running on your cluster.
Thanks!
-Sridhar
Hi Vladimir,
Yes, device caching is disabled. The ODSs in question use a
> separate DB/WAL device on flash. Could that be the cause of
> IOPS over-estimation?
>
2000 IOPS definitely looks high for a HDD. The in-built osd bench tool
writes to the backing device of the osd to determine the IOPS
Yes. Rotational drives can generally do 100-200IOPS (some outliers, of
course). Do you have all forms of caching disabled on your storage
controllers/disks?
On Tue, Sep 6, 2022 at 11:32 AM Vladimir Brik <
vladimir.b...@icecube.wisc.edu> wrote:
> Setting osd_mclock_force_run_benchmark_on_init to
Hello Vladimir,
I have noticed that our osd_mclock_max_capacity_iops_hdd
> varies widely for OSDs on identical drives in identical
> machines (from ~600 to ~2800).
>
The IOPS shouldn't vary widely if the drives are of similar age and running
the same workloads. The
