From: Sašo Kiselkov [mailto:skiselkov...@gmail.com]
Sent: Tuesday, September 11, 2012 9:52 AM
To: Dan Swartzendruber
Cc: 'James H'; email@example.com
Subject: Re: [zfs-discuss] Interesting question about L2ARC
On 09/11/2012 03:41 PM, Dan Swartzendruber wrote:
> LOL, I actually was unclear not you. I understood what you were
> saying, sorry for being unclear. I have 4 disks in raid10, so my max
> random read throughput is theoretically somewhat faster than the L2ARC
> device, but I never really do that intensive of reads.
But here's the kicker, prefetch is never random, it's always linear, so you
need measure prefetch throughput against the near linear throughput of your
disks. Your average 7k2 disk is capable of ~100MB/s in linear reads, so in a
pair-of-mirrors scenario (raid10) you get effectively in excess of 400MB/s
in prefetch throughput.
** True, and badly worded on my part. In theory, the 4 nearline SAS drives
could deliver 600MB/sec, but my path to the guests is maybe 3GB (this is all
running virtualized, so I can exceed gig/e speed.) The bottom line there is
that no amount of read effort by guests (via the hypervisor) is going to
come anywhere near the pool's capabilities.
> My point was: if a guest does read
> a bunch of data sequentially, that will trigger the prefetch L2ARC
> code path, correct?
No, when a client does a linear read, the initial buffer is random, so that
makes sense to serve from the l2arc - thus it is cached in the l2arc.
Subsequently ZFS detects that the client is likely to want more buffers, so
it will start prefetching the following blocks on the background. Then when
the client returns, they will receive the blocks from ARC. The result is
that the client wasn't latency constrained to process this block, so there's
no need to cache the subsequently prefetched blocks in l2arc.
*** Sorry, that's what I meant by 'the prefetch l2arc code path'. E.g. the
heuristics you referred to. It seems to me that if the client never wants
the prefetched block it was a waste to cache it, if he does, at worst, he'll
miss once, and then it will be cached, since it will have been a demand
> If so, I *do* want that cache in L2ARC, so that a return visit from
> that guest will hit as much as possible in the cache.
It will be in the normal ARC cache, however, l2arc is meant to primarily
accelerate the initial block hit (as noted above), not the subsequently
prefetched ones (which we have time to refetch from the main pool). This
covers most generic filesystem use cases as well as random-read heavy
workloads (such as databases, which rarely, if ever, do linear reads).
> One other
> thing (I don't think I mentioned this): my entire ESXi dataset is only
> like 160GB (thin provisioning in action), so it seems to me, I should
> be able to fit the entire thing in L2ARC?
Please try to post the output of this after you let it run on your dataset
for a few minutes:
$ arcstat.pl -f \
It should give us a good idea of the kind of workload we're dealing with and
why your L2 hits are so low.
*** Thanks a lot for clarifying how this works. Since I'm quite happy
having an SSD in my workstation, I will need to purchase another SSD :) I'm
wondering if it makes more sense to buy two SSDs of half the size (e.g.
128GB), since the total price is about the same?
zfs-discuss mailing list