Hi, I've been working on the problems of both slow performance and excessive file fragmentation, especially with regard to the RHEL7 kernel. I've created some patches (still forthcoming) to improve these things. For a long time, I thought the fragmentation issue was related to GFS2 not using its reservations efficiently. I wrote a patch that revises the calculations in function gfs2_write_calc_reserv, but it didn't seem to help. I also thought it might be releasing the reservations too early, but that turned out not to be the case. I've done extensive performance testing and determined that, even under the best circumstances (IOW by minimizing both inter and intra-node resource group contention), file fragmentation can be improved significantly by doubling the minimum block reservation size from 32 to 64 blocks. To test this, I did several two+ hour tests (real-life application) that uses lots of files in GFS2. After the test is complete, I totaled all the file extents (fragments) using filefrag.
Here are the results of five two-hour runs performed with new, revised calculations, and today's 32-block reservations: EXTENT COUNT FOR OUTPUT FILES = 444352 EXTENT COUNT FOR OUTPUT FILES = 468425 EXTENT COUNT FOR OUTPUT FILES = 472464 EXTENT COUNT FOR OUTPUT FILES = 482224 EXTENT COUNT FOR OUTPUT FILES = 481174 Here are the results of five runs performed with the old calculations and 64-block reservations: EXTENT COUNT FOR OUTPUT FILES = 365743 EXTENT COUNT FOR OUTPUT FILES = 398628 EXTENT COUNT FOR OUTPUT FILES = 404614 EXTENT COUNT FOR OUTPUT FILES = 401355 EXTENT COUNT FOR OUTPUT FILES = 384599 As you can see, by doubling the minimum reservation size, the files have about 20 percent fewer extents. This patch, therefore, doubles the minimum block reservations. Incidentally, if we don't take measures to minimize resource group contention (both inter and intra-node) the results are much worse. Here is the same test, done on a "stock" RHEL7 kernel: EXTENT COUNT FOR OUTPUT FILES = 826314 EXTENT COUNT FOR OUTPUT FILES = 791406 EXTENT COUNT FOR OUTPUT FILES = 735760 Patch description: The minimum block reservation size was 32 blocks. This patch doubles it to 64 blocks. Regards, Bob Peterson Red Hat File Systems Signed-off-by: Bob Peterson <rpete...@redhat.com> --- diff --git a/fs/gfs2/rgrp.h b/fs/gfs2/rgrp.h index 5d8f085..e2058a7 100644 --- a/fs/gfs2/rgrp.h +++ b/fs/gfs2/rgrp.h @@ -14,11 +14,9 @@ #include <linux/uaccess.h> /* Since each block in the file system is represented by two bits in the - * bitmap, one 64-bit word in the bitmap will represent 32 blocks. - * By reserving 32 blocks at a time, we can optimize / shortcut how we search - * through the bitmaps by looking a word at a time. + * bitmap, each 64-bit word in the bitmap represents 32 blocks. */ -#define RGRP_RSRV_MINBYTES 8 +#define RGRP_RSRV_MINBYTES 16 /* Reserve 64 blocks */ #define RGRP_RSRV_MINBLKS ((u32)(RGRP_RSRV_MINBYTES * GFS2_NBBY)) struct gfs2_rgrpd;
