> On Oct. 22, 2015, 9:48 a.m., Steven Hartland wrote: > > Quite a bit to go through in detail but unless I'm missing something in my > > initial pass my high level concern would be around TRIM causing IO > > saturation for leaf level vdevs. > > > > As you've mentioned TRIM requests are quite heavy compared with read / > > writes and different devices perform significantly differently due to > > internal differences on how they a process free requests, be that TRIM, > > UNMAP, ZERO etc. > > > > When we added ZFS TRIM support to FreeBSD one of the main issues was > > ensuring TRIM throughput was kept at a rate that the didn't significantly > > impact read / write IO's. > > > > In order to do this we changed the initial fixed TXG backlog, which seems > > to be what you have here, to be more dynamic. > > > > We now take into account the following factors per leaf vdev: > > 1. TXG max delay (trim_txg_delay - default 32) > > 2. Time max delay (trim_timeout - default 30s) > > 2. Max pending segments (trim_vdev_max_pending - default 10k) > > 3. Max interval between queue processing (trim_max_interval - default 1s) > > > > In addition we issue TRIM IOs using a different priority which has: > > 1. min active = 1 > > 2. max active = 64
Not sure there is much of an analogy between FreeBSD's implementation and this one here. The primary reason why I didn't try to do any kind of intelligent queueing and delaying is because it is largely unpredictable. Unlike read/write, which has a more-or-less linear load based on I/O size, TRIM is completely different. It can complete immediately or take several seconds, depending on whether the underlying device needs to unmap lots of LBAs or none. We don't know this, due to the underlying block remapping that many devices do and won't tell us about. I'm not opposed to thinking about ways of doing it, it's just that I haven't been able to come up with a suitably generic algorithm that doesn't suck on some devices and excels on others. If I could preuse your knowledge of FreeBSD's implementation though, I'd like to ask how FreeBSD *actually* issues TRIM/UNMAP commands. AFAICT, the upper layers just do single-extent BIO_DELETE to GELI and then some underlying magic happens which I'm not quite clear about. Most importantly, I'm very curious as to whether FreeBSD somehow intelligently re-aggregates issued extents and bunches them up into a single command, or issues one extent per command. I'm positive Linux does the latter, which is horridly inefficient and if FreeBSD does the same, it's where 90% of your TRIM/UNMAP performance suckage comes from. - Saso ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.csiden.org/r/263/#review849 ----------------------------------------------------------- On Oct. 27, 2015, 2:12 a.m., Saso Kiselkov wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.csiden.org/r/263/ > ----------------------------------------------------------- > > (Updated Oct. 27, 2015, 2:12 a.m.) > > > Review request for OpenZFS Developer Mailing List and Christopher Siden. > > > Repository: illumos-gate > > > Description > ------- > > Adds support for issuing abstract DKIOCFREE ioctls from ZFS and adds support > for translating these ioctls into SCSI UNMAP commands to sd. > This is an upstream of work by Nexenta. > > > Diffs > ----- > > usr/src/uts/common/fs/zfs/spa_misc.c > 6f255df85025d95675d5c4750ab762c7258b7b90 > usr/src/uts/common/Makefile.files c90a5c1773bf91d2437fcbfa90da81a48070ce70 > usr/src/pkg/manifests/system-header.mf > 4551ca095c0e25f47ddcb4e32d75c8eaaba0c1e4 > usr/src/uts/common/sys/sysevent/eventdefs.h > 9c6907a08af65665cdb09588c3b0ef89f087d70c > usr/src/uts/common/sys/scsi/targets/sddef.h > 39c0ed9d0fb2c2d6c20fa793c3b5f9168a844552 > usr/src/uts/common/sys/fs/zfs.h bc9f057dd1361ae73a12375515abacd0fed820d2 > usr/src/uts/common/sys/dkioc_free_util.h PRE-CREATION > usr/src/uts/common/sys/dkio.h a5b0c312f9df59a7171778411ccaff654c5b27e8 > usr/src/uts/common/sys/Makefile 39288d5cc0dd2b025f8e8661664f3e77c9fa9272 > usr/src/uts/common/os/dkioc_free_util.c PRE-CREATION > usr/src/uts/common/io/scsi/targets/sd.c > ae1e7e0fc3e51957dd66158c960251550ed9890d > usr/src/uts/common/io/comstar/lu/stmf_sbd/stmf_sbd.h > efbc7268ea7aab11b1d726551058d38e71bf376d > usr/src/uts/common/io/comstar/lu/stmf_sbd/sbd_scsi.c > cb6e115fe949145d39865333353ef50baf49c7da > usr/src/uts/common/io/comstar/lu/stmf_sbd/sbd.c > e8a4b131380376ea0dc028c01a6350cb438ecff2 > usr/src/uts/common/fs/zfs/zvol.c 585500bbac86d23fb82eea7ed5e3e7d3970a25e5 > usr/src/uts/common/fs/zfs/zio.c 4a3dafac9f7f391a9f16c06932420f655dde61f0 > usr/src/uts/common/fs/zfs/zfs_ioctl.c > c863cbd399a88dd70edcd685fde6dadffbac3ff7 > usr/src/uts/common/fs/zfs/vdev_root.c > a5442a55eb5a23fbd6f5cf76f8b348670ad2e621 > usr/src/uts/common/fs/zfs/vdev_raidz.c > 085d1250a1ad7bef0ab4a1a2e92b90eb308ee455 > usr/src/uts/common/fs/zfs/vdev_missing.c > 228757334234d241f980058397438d3a80716dcf > usr/src/uts/common/fs/zfs/vdev_mirror.c > 8749e539f46682f3bb073fbf119004c6ecc64177 > usr/src/uts/common/fs/zfs/vdev_label.c > c9b0641cfd6318b686e13cd3f2ada4618f028702 > usr/src/uts/common/fs/zfs/vdev_file.c > 5dfc331d20a480047e44a59bb504319b34440e17 > usr/src/uts/common/fs/zfs/vdev_disk.c > ed4a8b773bf4d9a5f19a9b679bd393729ff529cc > usr/src/uts/common/fs/zfs/vdev.c 1c57fce4dcee909b164353181dcd8e2a29ed7946 > usr/src/uts/common/fs/zfs/sys/zio_impl.h > 08f820103e823681031100c8b2f65f8661e8293e > usr/src/uts/common/fs/zfs/sys/zio.h > 877e2839bf804ca3625c907367fed4a06744ea52 > usr/src/uts/common/fs/zfs/sys/vdev_impl.h > 17a18a319934908190409a0eeb5b18ff83b9e001 > usr/src/uts/common/fs/zfs/sys/vdev.h > 08ce54b18c78a6f5b5a59658e46655e66007aa5c > usr/src/uts/common/fs/zfs/sys/spa_impl.h > 441800198215e53fec306f6e5246642f9076d8e4 > usr/src/uts/common/fs/zfs/sys/spa.h > 7ac78390338a44f7b7658017e1ae8fcc9beb89d6 > usr/src/uts/common/fs/zfs/sys/range_tree.h > 9f3ead537165f3a7b8c52fe58eedef66c1b1952e > usr/src/uts/common/fs/zfs/sys/metaslab_impl.h > 27a53b515fbc48ab5b200e88259df91cb6effe19 > usr/src/uts/common/fs/zfs/sys/metaslab.h > b3b9374c779f2460ca001577a84ade28210cc7ce > usr/src/uts/common/fs/zfs/spa_config.c > 47bb59590893cb72ca7f2cee397566c7e466d6d4 > usr/src/uts/common/fs/zfs/spa.c 95a6b0fae7760e8a1e8cfc1e657dc22fd9ef3245 > usr/src/uts/common/fs/zfs/range_tree.c > 6422fd1c1fa6ecf2a9283fefeabca772f6b0a76a > usr/src/uts/common/fs/zfs/metaslab.c > 852534eff8bd8cb4351fe720e0b71bef255bbfaa > usr/src/uts/common/fs/zfs/dsl_scan.c > 53902793ef5ccce6e4c79d013df36bf381677ee9 > usr/src/man/man1m/zpool.1m fbfd39357930d7550226ba7ca180042b96c28c1a > usr/src/lib/libzpool/common/sys/zfs_context.h > 9e4d8ed0b8ec42be75bb93f44602ac99e907cf00 > usr/src/lib/libzpool/Makefile.com da5da5d93682faf90a71e329f6345409513496dc > usr/src/lib/libzfs/common/mapfile-vers > dc72ab001049757ae5b6eac56716dbc954046f3c > usr/src/lib/libzfs/common/libzfs_pool.c > 3c992951793d2ef5d63c31348ec1b0754d5d6964 > usr/src/lib/libzfs/common/libzfs.h 1ebf520297365b423fce4635646533804d1eaad9 > usr/src/common/zfs/zpool_prop.c 4d906b02bc02e80a7b0aae7af66898b6e5d1ae79 > usr/src/cmd/zpool/zpool_main.c 18af5a2763039694d1c6acddb89f26aa7a2b12d1 > > Diff: https://reviews.csiden.org/r/263/diff/ > > > Testing > ------- > > Run on assortment of raidz, mirrors and straight vdevs. > > > Thanks, > > Saso Kiselkov > >
_______________________________________________ developer mailing list developer@open-zfs.org http://lists.open-zfs.org/mailman/listinfo/developer
