On Fri, May 18, 2012 at 03:05:09AM +0400, Jim Klimov wrote:
>   While waiting for that resilver to complete last week,
> I caught myself wondering how the resilvers (are supposed
> to) work in ZFS?

The devil finds work for idle hands... :-)

>   Based on what I see in practice and read in this list
> and some blogs, I've built a picture and would be grateful
> if some experts actually familiar with code and architecture
> would say how far off I guessed from the truth ;)

Well, I'm not that - certainly not on the code.  It would probably be
best (for both of us) to spend idle time looking at the code, before
spending too much on speculation. Nonetheless, let's have at it! :)

>   Ultimately I wonder if there are possible optimizations
> to make the scrub process more resembling a sequential
> drive-cloning (bandwidth/throughput-bound), than an
> IOPS-bound random seek thrashing for hours that we
> often see now, at least on (over?)saturated pools.

The tradeoff will be code complexity and resulting fragility. Choose
wisely what you wish for.

> This may possibly improve zfs send speeds as well.

Less likely, that's pretty much always going to have to go in txg

>   First of all, I state (and ask to confirm): I think
> resilvers are a subset of scrubs, in that:
> 1) resilvers are limited to a particular top-level VDEV
> (and its number is a component of each block's DVA address)
> and
> 2) when scrub finds a block mismatching its known checksum,
> scrub reallocates the whole block anew using the recovered
> known-valid data - in essence it is a newly written block
> with a new path in BP tree and so on; a resilver expects
> to have a disk full of known-missing pieces of blocks,
> and reconstructed pieces are written on the resilvering
> disk "in-place" at an address dictated by the known DVA -
> this allows to not rewrite the other disks and BP tree
> as COW would otherwise require.

No. Scrub (and any other repair, such as for errors found in the
course of normal reads) rewrite the reconstructed blocks in-place: to
the original DVA as referenced by its parents in the BP tree, even if
the device underneath that DVA is actually a new disk.

There is no COW. This is not a rewrite, and there is no original data
to preserve, this is a repair: making the disk sector contain what the
rest of the filesystem tree 'expects' it to contain. More specifically,
making it contain data that checksums to the value that block pointers
elsewhere say it should, via reconstruction using redundant
information (same DVA on a mirror/RAIDZ recon, or ditto blocks at
different DVAs found in the parent BP for copies>1, including metadata)

BTW, if a new BP tree was required to repair blocks, we'd have
bp-rewrite already (or we wouldn't have repair yet).

>   Other than these points, resilvers and scrubs should
> work the same, perhaps with nuances like separate tunables
> for throttling and such - but generic algorithms should
> be nearly identical.
> Q1: Is this assessment true?

In a sense, yes, despite the correction above.  There is less
difference between these cases than you expected, so they are nearly
identical :-)

>   So I'll call them both a "scrub" below - it's shorter :)

Call them all repair.

The difference is not in how repair happens, but in how the need for a
given sector to be repaired is discovered.

Let's go over those, and clarify terminology, before going through the
rest of your post:

 * Normal reads: a device error or checksum failure triggers a

 * Scrub: Devices may be fine, but we want to verify that and fix any
   errors. In particular, we want to check all redundant copies.

 * Resilver: A device has been offline for a while, and needs to be
   'caught up', from its last known-good TXG to current.

 * Replace: A device has gone, and needs to be completely

Scrub is very similar to normal reads, apart from checking all copies
rather than serving the data from whichever copy successfully returns
first. Errors are not expected, are counted and repaired as/if found.

Resilver and Replace are very similar, and the terms are often used
interchangably. Replace is essentially resilver with a starting TXG of
0 (plus some labelling). In both cases, an error is expected or
assumed from the device in question, and repair initiated
unconditionally (and without incrementing error counters). 

You're suggesting an assymetry between Resilver and Replace to exploit
the possibile speedup of sequential access; ok, seems attractive at
first blush, let's explore the idea.

>   Now, as everybody knows, at least by word-of-mouth on
> this list, the scrub tends to be slow on pools with a rich
> life (many updates and deletions, causing fragmentation,
> with "old" and "young" blocks intermixed on disk), more
> so if the pools are quite full (over about 80% for some
> reporters). This slowness (on non-SSD disks with non-zero
> seek latency) is attributed to several reasons I've seen
> stated and/or thought up while pondering. The reasons may
> include statements like:
> 1) "Scrub goes on in TXG order".

Yes, it does, approximately. More below.

> If it is indeed so - the system must find older blocks,
> then newer ones, and so on. IF the block-pointer tree
> starting from uberblock is the only reference to the
> entirety of the on-disk blocks (unlike say DDT)

(aside: it is. The DDT is not special in this sense, because to find
the DDT you have to follow the bp tree too.)

> then
> this tree would have to be read into memory and sorted
> by TXG age and then processed.
> From my system's failures I know that this tree would
> take about 30Gb on my home-NAS box with 8Gb RAM, and
> the kernel crashes the machine by depleting RAM and
> not going into swap after certain operations (i.e.
> large deletes on datasets with enabled deduplication).
> That was discussed last year by me, and recently by
> other posters.
> Since the scrub does not do that and does not even
> press on RAM in a fatal manner, I think this "reason"
> is wrong.

Well, your observations and analysis of what scrub is not doing are
correct and sound.. :-)

> I also fail to see why one would do that
> processing ordering in the first place - on a fairly
> fragmented system even the blocks from "newer" TXGs
> do not necessarily follow those from the "previous"
> ones.

You're thinking too much about the on-disk ordering of sector
numbers.  Understandable, since you're trying to find a way to do
sequential repair.  

For now, let's just say that going in TXG order is the easiest way to
iterate over the disk and be sure to get all live data, without doing
other complicated and memory/IO-intensive sorts. Again, we'll come back
to this.  

> What this rumour could reflect, however, is that a scrub
> (or more importantly, a resilver) are indeed limited by
> the "interesting" range of TXGs, such as picking only
> those blocks which were written between the last TXG that
> a lost-and-reconnected disk knew of (known to the system
> via that disk's stale uberblock), and the current TXG
> at the moment of its reconnection. Newer writes would
> probably land onto all disks anyway, so a resilver has
> only to find and fix those missing TXG numbers.

Yes, for resilver this is spot on, as above.

> In my problematic system however I only saw full resilvers
> even after they restarted numerously... This may actually
> support the idea that scrubs are NOT txg-ordered, otherwise
> a regularly updated tabkeeping attribute on the disk (in
> uberblock?) would note that some TXGs are known to fully
> exist on the resilvering drive - and this is not happening.

Now you have two problems:

 * confusing scrub (as a way of checking and possibly triggering
   repair) with resilver (known need to repair).
 * older code: in newer code there is better bookkeeping, at least for
   scrub, that allows a resume (after, say a reboot) from where it
   left off.  I'm not sure about resilver here, though (and note the
   complexity with the optimisation of 'new writes' past the offline
   window, above).

> 2) "Scrub walks the block-pointer tree".

yes, it does. It's essentially the same as the previous point, though:
scrub walks the bp tree in txg order.

> That seems like a viable reason for lots of random reads
> (hitting the IOPS barrier). 

Yep.  We're getting closer to the real reason here, but let's play it
out in full as we go.

> It does not directly explain
> the reports I think I've seen about L2ARC improving scrub
> speeds and system responsiveness - although extra caching
> takes the repetitive load off the HDDs and leaves them
> some more timeslices to participate in scrubbing (and
> *that* should incur reads from disks, not caches).

If L2ARC indeed helps, it will surely be mostly to do with improving
responsiveness on other reads and freeing up the disks to do scrubs. 

> On an active system, block pointer entries are relatively
> short-lived, with whole branches of a tree being updated
> and written in a new location upon every file update.
> This image is bound to look like good cheese after a while
> even if the writes were initially coalesced into few IOs.

You might be surprised, you probably have more long-lived data than
you thought, especially with snapshots in place.  The full metadata
bp tree path to that old data is also retained.

Note also the corollary: whenever data is COW'd, the full metadata
path is also COW'd (possibly rolled up together with other updates in
the same TXG).  What that means is that, to read data for a new TXG as
you progress in a resilver, replace or scrub, you have to read all new

> 3) "If there are N top-level VDEVs in a pool, then only
> the one with the resilvering disk would be hit for
> performance" - not quite true, because pieces of the
> BPtree are spread across all VDEVs. The one resilvering
> would get the most bulk traffic, when DVAs residing on
> it are found and userdata blocks get transferred, but
> random read seeks caused by the resilvering process
> should happen all over the pool.

Not sure what this one means and I think it's mostly false for the
reason you state.  

Either resilvering or replacing, the disk is mostly getting writes -
and cachable writes at that - from this activity. For resilver
especially, it might see reads for other concurrent activity.

The IOPS limitation is for seeks necessary to satisfy reads, mostly
from other disks, to provide data for reconstruction.  As noted above,
if a disk is being resilvered for TXG n, it won't have any of the
metadata for that TXG either, so won't really be servicing any reads.

> Q2: Am I correct with the interpretation of statements 1-3?

Not quite, as discussed above.

Let's go over the scrub case in detail (resilver being a txg
window-limited variant, and both resilver and replace enabling
different error reporting logic).

 * Every meta/data block in the disk was written in a given TXG.
 * Every meta/data block is reachable by a path through the bp tree,
   from the root at the close of that TXG, down through however many
   indirect levels are needed. 
 * For every later TXG while the data remains current, the new root
   and top few nodes in the tree will change, (due to other writes) but
   those upper nodes will refer to the same subtree below the point of 
   divergence caused by those later writes. In other words, each TXG
   assembles a bp tree from a new root, and reuses subtrees from the
   previous TXG where no changes have been made.
 * Snapshots are simply additional references to old (filesystem) root
   bp's, as a way to keep that subtree live.

And here's the kicker for any attempt at LBA-sequential repair:

 * The checksum for a given block, that allows it to be verified, is
   stored in the bp that refers to it.

So, if reading blocks sequentially, you can't verify them. You don't
know what their checksums are supposed to be, or even what they
contain or where to look for the checksums, even if you were prepared
to seek to find them.  This is why scrub walks the bp tree.

When doing a scrub, you start at the root bp and walk the tree, doing
reads for everything, verifying checksums, and letting repair happen
for any errors. That traversal is either a breadth-first or
depth-first traversal of the tree (I'm not sure which) done in TXG

When you're done with that bp tree, the pool has almost certainly
moved on with new TXG's. Get the new root bp, and do the traversal
again. This time, any bp with a birth time equal or older to the TXG
you previously finished has already been verified, including the
entire subtree below, and so can be skipped.  This is why scrub walks
in TXG order.  It's also why the disk access is in 'approximate TXG
order', as you'll sometimes see the more pedantic commenters state.

Note that there can be a lot of fanout in the tree; don't make the
mistake of thinking that the directories and filesystems you see are
the tree in question; the ZPL is a layer on top of the ZAP object

> One optimization that could take place here would be to
> store some of the BPs' ditto copies in compact locations
> on disk (not all over it evenly), albeit maybe hurting
> the write performance. This way a resilver run, or even
> a scrub or zfs send, might be like a vdev-prefetch - a
> scooping read of several megabytes worth of blockpointers
> (this would especially help if the whole tree would fit
> in RAM/L2ARC/swap), then sorting out the tree or its major
> branches. The benefit would be little mechanical seeking
> for lots of BP data. This might possibly require us to
> invalidate the freed BP slots somehow as well :\
> In case of scrubs, where we would have to read in all of
> the allocated blocks from the media to test it, this would
> let us schedule a sequential read of the drives userdata
> while making sense of the sectors we find (as particular
> zfs blocks).
> In case of resilvering - this would let us find DVAs of
> blocks in the interesting TLVDEV and in the TXG range and
> also schedule huge sequential reads instead of random
> seeking.
> In case of zfs send, this would help us pick out the
> TXG-limited ranges of the blocks for a dataset, and
> again schedule the sequential reads for userdata (if any).
> Q3: Does the IDEA above make sense - storing BP entries
> (one of the ditto blocks) in some common location on disk,
> so as to minimize mechanical seeks while reading much of
> the BP tree?

It's not going to help a scrub, since that reads all of the ditto
block copies, so bunching just one copy isn't useful. It might
potentially help metadata-heavy activities that don't touch the data,
like find(1), at the expense of several other issues, at least some of
which you note.

That said, there are always opportunities for tweaks and improvements
to the allocation policy, or even for multiple allocation policies
each more suited/tuned to specific workloads if known in advance.

> It seems possible to enable defragmentation of the BP tree
> (those ditto copies that are stored together) by just
> relocating the valid ones in correct order onto a free
> metaslab.

"Just" is such a four-letter word.

If you move a bp, you change its DVA. Which means that the parent bp
pointing to it needs to be updated and rewritten, and its parents as
well. This is new, COW data, with a new TXG attached -- but referring
to data that is old and has not been changed. You just broke snapshots
and scrub, at least.

This is bp rewrite, or rather, why bp-rewrite is hard.

> It seems that ZFS keeps some free space for
> passive defrag purposes anyway - why not use it actively?
> Live migration of blocks like this seems to be available
> with scrub's repair of the mismatching blocks. 

This gets back to the misunderstanding (way) above.  Repair is not
COW; repair is repairing the disk block to the original, correct

> However,
> here some care should be taken to take into account that
> the parent blockpointers would also need to be reallocated
> since the childrens' checksums would change - so the whole
> tree/branch of reallocations would have to be planned and
> written out in sequential order onto the spare free space.

And more complexities, since you want this done on a live pool. 

> Overall, if my understanding somewhat resembles how things
> really are, these ideas may help create and maintain such
> layout of metadata that it can be bulk-read, which is IMHO
> critical for many operations as well as to shorted recovery
> windows when resilvering disks.
> Q4: I wonder if similar (equivalent) solutions are already
> in place and did not help much? ;)

At least scrub does more book-keeping in more recent code and will
avoid restarts and rework.

I would like to see a replace variant that signals that at least some
of the data on the disk may already be valid, so it could potentially
be used in reconstruction when multiple disks have errors.


Attachment: pgp4NWmw3Brpx.pgp
Description: PGP signature

zfs-discuss mailing list

Reply via email to