Hi,
On 12/06/15 20:50, Bob Peterson wrote:
- Original Message -
Hi,
On 09/06/15 15:45, Bob Peterson wrote:
- Original Message -
Hi,
On 05/06/15 15:49, Bob Peterson wrote:
Hi,
This patch allows the block allocation code to retain the buffers
for the resource groups so they don't need to be re-read from buffer
cache with every request. This is a performance improvement that's
especially noticeable when resource groups are very large. For
example, with 2GB resource groups and 4K blocks, there can be 33
blocks for every resource group. This patch allows those 33 buffers
to be kept around and not read in and thrown away with every
operation. The buffers are released when the resource group is
either synced or invalidated.
The blocks should be cached between operations, so this should only be
resulting in a skip of the look up of the cached block, and no changes
to the actual I/O. Does that mean that grab_cache_page() is slow I
wonder? Or is this an issue of going around the retry loop due to lack
of memory at some stage?
How does this interact with the rgrplvb support? I'd guess that with
that turned on, this is no longer an issue, because we'd only read in
the blocks for the rgrps that we are actually going to use?
Steve.
Hi,
If you compare the two vmstat outputs in the bugzilla #1154782, you'll
see no significant difference in memory usage nor cpu usage. So I assume
the page lookup is the slow part; not because it's such a slow thing
but because it's done 33 times per read-reference-invalidate (33 pages
to look up per rgrp).
Regards,
Bob Peterson
Red Hat File Systems
Thats true, however, as I understand the problem here, the issue is not
reading in the blocks for the rgrp that is eventually selected to use,
but the reading in of those blocks for the rgrps that we reject, for
whatever reason (full, or congested, or whatever). So with rgrplvb
enabled, we don't then read those rgrps in off disk at all in most cases
- so I was wondering whether that solves the problem without needing
this change?
Ideally I'd like to make the rgrplvb setting the default, since it is
much more efficient. The question is how we can do that and still remain
backward compatible? Not an easy one to answer :(
Also, if the page lookup is the slow thing, then we should look at using
pagevec_lookup() to get the pages in chunks rather than doing it
individually (and indeed, multiple times per page, in case of block size
less than page size). We know that the blocks will always be contiguous
on disk, so we should be able to send down large I/Os, rather than
relying on the block stack to merge them as we do at the moment, which
should be a further improvement too,
Steve.
Hi,
The rgrplvb mount option only helps if the file system is using lock_dlm.
For lock_nolock, it's still just as slow because lock_nolock has no knowledge
of lvbs. Now, granted, that's an unusual case because GFS2 is normally used
with lock_dlm.
That sounds like a bug... it should work in the same way, even with
lock_nolock.
I like the idea of making rgrplvb the default mount option, and I don't
see a problem doing that.
The issue is that it will not be backwards compatible. We really need a
way to at least easily detect if someone has a mixed cluster (some nodes
with rgrplvb enabled, and some without) otherwise it might be confusing
when we get odd reports of allocations failing, even when there appears
to be free space.
We need to treat what we put into LVBs in the same way as we treat the
on-disk format in terms of
backwards compatibility.
I think the rgrplvb option should be compatible with this patch, but
I'll set up a test environment in order to test that they work together
harmoniously.
I also like the idea of using a pagevec for reading in multiple pages for
the rgrps, but that's another improvement for another day. If there's
not a bugzilla record open for that, perhaps we should open one.
Regards,
Bob Peterson
Red Hat File Systems
If we have rgrplvb, then we my not need this patch, since we will not be
looking up the rgrp's blocks as often. So we should see the benefit just
by turning that on I think... at least it would be good to see whether
there is any performance difference there. In cases where we have nodes
competing for the rgrps, then the blocks will not be cached anyway, so
we will gain no benefit from this patch, since we'll have to read the
blocks anyway, hence my thought that speeding up the lookup is the way
to go, since it will give the benefit for more different cases - both
when the rgrps blocks are cached and uncached,
Steve.
