Re: [Cluster-devel] [PATCH 00/19] gfs2-utils: Introduce extent allocation and speed up journal creation

2014-09-03 Thread Steven Whitehouse

Hi,

Aside from the question on patch 6, the other patches all look good,

Steve.

On 02/09/14 13:07, Andrew Price wrote:

Hi,

This patch set introduces extent allocation to libgfs2 and adds functions which
decouple file creation, allocation and writing so that mkfs.gfs2 can be
re-worked to write journals and resource groups sequentially.

With these patches, mkfs.gfs2 typically takes around 20% of the time that it
did before in my tests.  The main speed-up has been from the journal data
allocation functions not having to re-read and write a resource group for each
block allocated as it did before (this was a performance regression introduced
by previous memory footprint improvement patches, hence the significant perf
improvement).  Journals now each occupy an extent spanning an entire resource
group specifically sized for the journal, and the resource group headers are
written only once, after the journal blocks have been allocated in the
in-memory bitmaps. Resource groups are still only kept in memory for as long as
they are needed so peak memory usage should be largely unchanged.

One thing to note is that, with these patches, the root and master inodes are
no longer the first objects in the first resource group. The master inode is
written in the first free block after the journals and then the other metafs
structures are placed. The root directory inode is then finally created. This
is not a format change but it may cause some confusion after years of expecting
the root and master inodes to be at certain addresses so I thought it worth
mentioning.

Coverity and valgrind are happy about these patches and I've encountered no
problems after various tests which mount the fs.

Cheers,
Andy

Andrew Price (19):
   libgfs2: Keep a pointer to the sbd in lgfs2_rgrps_t
   libgfs2: Move bitmap buffers inside struct gfs2_bitmap
   libgfs2: Fix an impossible loop condition in gfs2_rgrp_read
   libgfs2: Introduce struct lgfs2_rbm
   libgfs2: Move struct _lgfs2_rgrps into rgrp.h
   libgfs2: Add functions for finding free extents
   tests: Add unit tests for the new extent search functions
   libgfs2: Ignore an empty rgrp plan if a length is specified
   libgfs2: Add back-pointer to rgrps in lgfs2_rgrp_t
   libgfs2: Const-ify the parameters of print functions
   libgfs2: Allow init_dinode to accept a preallocated bh
   libgfs2: Add extent allocation functions
   libgfs2: Add support for allocating entire rgrp headers
   libgfs2: Write file metadata sequentially
   libgfs2: Fix alignment in lgfs2_rgsize_for_data
   libgfs2: Handle non-zero bitmaps in lgfs2_rgrp_write
   libgfs2: Add a speedier journal data block writing function
   libgfs2: Create jindex directory separately from journals
   mkfs.gfs2: Improve journal creation performance

  .gitignore  |   3 +-
  gfs2/convert/gfs2_convert.c |  49 +++--
  gfs2/edit/journal.c |   6 +-
  gfs2/fsck/fs_recovery.c |   2 +-
  gfs2/fsck/initialize.c  |  27 +--
  gfs2/fsck/metawalk.c|  10 +-
  gfs2/fsck/pass5.c   |   9 +-
  gfs2/fsck/rgrepair.c|  14 +-
  gfs2/fsck/util.c|   2 +-
  gfs2/libgfs2/Makefile.am|   2 +-
  gfs2/libgfs2/fs_bits.c  |  10 +-
  gfs2/libgfs2/fs_geometry.c  |   6 +-
  gfs2/libgfs2/fs_ops.c   | 184 ++---
  gfs2/libgfs2/libgfs2.h  |  50 +++--
  gfs2/libgfs2/ondisk.c   |  26 +--
  gfs2/libgfs2/rgrp.c | 491 
  gfs2/libgfs2/rgrp.h |  50 +
  gfs2/libgfs2/structures.c   | 103 +-
  gfs2/mkfs/main_grow.c   |   4 +-
  gfs2/mkfs/main_mkfs.c   | 155 ++
  tests/Makefile.am   |  33 ++-
  tests/check_rgrp.c  | 143 +
  tests/libgfs2.at|   8 +-
  23 files changed, 1113 insertions(+), 274 deletions(-)
  create mode 100644 gfs2/libgfs2/rgrp.h
  create mode 100644 tests/check_rgrp.c





Re: [Cluster-devel] [PATCH 00/19] gfs2-utils: Introduce extent allocation and speed up journal creation

2014-09-02 Thread Bob Peterson
- Original Message -
 One thing to note is that, with these patches, the root and master inodes are
 no longer the first objects in the first resource group. The master inode is
 written in the first free block after the journals and then the other metafs
 structures are placed. The root directory inode is then finally created. This
 is not a format change but it may cause some confusion after years of
 expecting
 the root and master inodes to be at certain addresses so I thought it worth
 mentioning.

Hi,

I know that in fsck.gfs2, in initialize.c, it plays some games trying to find
and repair damaged system dinodes. For example, it looks for a missing master
directory by looking for no_formal_ino==2 for example. So I'd be very cautious
and check to make sure these repairs still work properly. In the past, I've
done a for loop, wiping out the first X blocks of the file system, running
fsck.gfs2, and seeing if it can properly repair it.

Another concern is gfs2_convert. I don't know if it makes any assumptions
about the master directory, but it's much less likely. I think it just
assumes the file system is healthy. But fsck.gfs2 is a concern.

Regards,

Bob Peterson
Red Hat File Systems