On Thu, Aug 03, 2017 at 05:08:47PM +0000, Eric Robinson wrote:
> For anyone else who has this problem, we have reduced the time
> required to trim a 1.3TB volume from 3 days to 1.5 minutes.
> Initially, we used mdraid rto build a raid0 array with a 32K chunk
> size. We initialized it as a drbd disk, synced it, built an lvm
> logical volume on it, and created an ext4 filesystem on the volume.
> Creating the filesystem and tripping it took 3 days (each time, every
> time, across multiple tests).
> When running lsblk -D, we noticed that the DISC-MAX value for the
> array was only 32K, compared to 4GB for the SSD drive itself. We also
> noticed that the number matched the chunk size. We theorized that the
> small DISC-MAX value was responsible for the slow trim rate across the
> DRBD link. We deleted the array and built a new one with a 4MB chunk
> size. The DISC-MAX value changed 4MB, which is the max selectable
> chunk size (but still way below the other DISC-MAX values shown in
> lsblk -D). We realized that, when using mdadm, the DISK-MAX value ends
> up matching the array chunk size.
> Instead of using mdadm to build the array, we used LVM to create a
> striped logical volume and made that the backing device for drbd. Then
> lsblk -D showed a DISC-MAX size of 128MB. Creating an ext4 filesystem
> on it and trimming only took 1.5 minutes (across multiple tests).
> Somebody knowledgeable may be able to explain how DISC-MAX affects the
> trim speed, and why the DISC-MAX value is different when creating the
> array with mdadm versus lvm.
Usually, there will only be one single discard / trim / unmap
request in flight.
Earlier, you said a rtt of ~ 1ms.
Assuming for a moment that that was the only source of latency,
and if we ignore any other possible overhead:
io-depth of 1, 1000 requests per second, 32k per request,
you will max out at ~ 32 MByte per second,
io-depth of 1, 1000 requests per second, 128 MB per request,
you will max out at ~ 128 GB per second,
which means you hit some other bottleneck much earlier
(the discard bandwidth of the backing storage...)
Note that DRBD 9.0.8 still has a problem with discards larger than 4 MB,
though (will hit protocoll error, disconnect, and reconnect).
That is already fixed in git, 9.0.9rc1 has that fixed.
(8.4.10 also works fine there)
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
DRBD® and LINBIT® are registered trademarks of LINBIT
please don't Cc me, but send to list -- I'm subscribed
drbd-user mailing list