Hi, On October 8, I posted a GFS2 patch that greatly reduced inter-node contention for resource group glocks. The patch was called: "GFS2: Set of distributed preferences for rgrps". It implemented a new scheme whereby each node in a cluster tries to "keep to itself" for allocations. This is not unlike GFS1, which has a different scheme.
Although the patch sped up GFS2 performance in general, it also caused more file fragmentation, because each node tended to focus on a smaller subset of resource groups. Here are run times and file fragmentation extent counts for my favorite customer application, using a STOCK RHEL7 kernel (no patches): Run times: Run 1 time: 2hr 40min 33sec Run 2 time: 2hr 39min 52sec Run 3 time: 2hr 39min 31sec Run 4 time: 2hr 33min 57sec Run 5 time: 2hr 41min 6sec Total file extents (File fragmentation): EXTENT COUNT FOR OUTPUT FILES = 744708 EXTENT COUNT FOR OUTPUT FILES = 749868 EXTENT COUNT FOR OUTPUT FILES = 721862 EXTENT COUNT FOR OUTPUT FILES = 635301 EXTENT COUNT FOR OUTPUT FILES = 689263 The times are bad and the fragmentation level is also bad. If I add just the first patch, "GFS2: Set of distributed preferences for rgrps" you can see that the performance improves, but the fragmentation is worse (I only did three iterations this time): Run times: Run 1 time: 2hr 2min 47sec Run 2 time: 2hr 8min 37sec Run 3 time: 2hr 10min 0sec Total file extents (File fragmentation): EXTENT COUNT FOR OUTPUT FILES = 1011217 EXTENT COUNT FOR OUTPUT FILES = 1025973 EXTENT COUNT FOR OUTPUT FILES = 1070163 So the patch improved performance by 25 percent, but file fragmentation is 30 percent worse. Some of this is undoubtedly due to the SAN array buffering, hiding our multitude of sins. But not every customer will have this quality of SAN. So it's important to reduce the fragmentation as well, so that some people are helped while others are hurt by the patch. Toward this end, I devised three relatively simple patches that greatly reduce file fragmentation. With all four patches, the numbers are as follows: Run times: Run 1 time: 2hr 5min 46sec Run 2 time: 2hr 10min 15sec Run 3 time: 2hr 8min 4sec Run 4 time: 2hr 9min 27sec Run 5 time: 2hr 6min 15sec Total file extents (File fragmentation): EXTENT COUNT FOR OUTPUT FILES = 330276 EXTENT COUNT FOR OUTPUT FILES = 358939 EXTENT COUNT FOR OUTPUT FILES = 375374 EXTENT COUNT FOR OUTPUT FILES = 383071 EXTENT COUNT FOR OUTPUT FILES = 369269 As you can see, with this combination of four patches, the run times are good as well as the file fragmentation levels. The file fragmentation is about twice as good as the stock kernel, and significantly better (almost three times better) than with the first patch alone. This patch set includes all four patches. Bob Peterson (4): GFS2: Set of distributed preferences for rgrps GFS2: Make block reservations more persistent GFS2: Only increase rs_sizehint GFS2: If we use up our block reservation, request more next time fs/gfs2/file.c | 10 ++------ fs/gfs2/incore.h | 2 ++ fs/gfs2/lock_dlm.c | 2 ++ fs/gfs2/ops_fstype.c | 1 + fs/gfs2/rgrp.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++---- 5 files changed, 71 insertions(+), 13 deletions(-) -- 1.9.3
