On Thu, Jul 29, 2010 at 8:54 PM, Andreas Dilger
<[email protected]> wrote:
> On 2010-07-29, at 04:47, Daire Byrne wrote:
>> I was wondering if it is possible to have the client completely cache
>> a recursive listing of a lustre filesystem such that on a second run
>> it doesn't have to talk to the MDT again? Taking the simplest case
>> where I only have one client that is browsing a million file tree
>> (say), I would expect that once the ldlm has cached the locks
>> (lru_size) then a second recursive scan (find, ls -R) shouldn't need
>> to talk to the MDT/OST again. But this is not the case probably
>> because a recursive scan needs to do a open() and getdents() on each
>> directory it finds.
>
> The getdents() calls can be returned from the client-side cache, it is only
> the open() that needs to go to the MDS. Lustre actually does support
> client-side open cache, but it is currently only used by NFS servers (which,
> sadly, opens and closes the file for every single write operation on a file).
Ah yes... that makes sense. I recall the opencache gave a big boost in
performance for NFS exporting but I wasn't sure if it had become the
default. I haven't been keeping up to date with Lustre developments.
> I know Oleg has at times discussed enabling the open cache on the client for
> regular filesystem access, but I don't know the tweak needed for this
> offhand. I know in the past we didn't do this because there was extra DLM
> locking overhead for cancelling the open lock, but with the DLM lock cancel
> batching that may not be as big a performance hit.
>
> It wouldn't be a bad idea to start with a /proc tuneable or "-o openlock"
> mount option that selectively allows open cache per client mount, so that
> performance testing can be done. After that we can decide whether this is
> only good for specific workloads and bad for others, or if it is an
> improvement for most workloads and should be enabled by default.
So just as a quick and dirty check of the opencache I did a "find" on
a remote client directly using the lustre client and also through an
NFS gateway client (in this case running on the MDS).
find /mnt/lustre (not cached) = 51 secs
find /mnt/lustre (cached = 22 secs
find /mnt/nfs (not cached) = 127 secs
find /mnt/nfs (cached) = 15 secs.
So even with the metadata going over NFS the opencache in the client
seems to make quite a difference (I'm not sure how much the NFS client
caches though). As expected I see no mdt activity for the NFS export
once cached. I think it would be really nice to be able to enable the
opencache on any lustre client. A couple of potential workloads that I
can think of that would benefit are WAN clients and clients that need
to do mainly metadata (e.g. scanning the filesystem, rsync --link-dest
hardlink snapshot backups). For the WAN case I'd be quite interested
to see what the overhead of the lock cancellation would be like for a
busy filesystem. I suppose we can already test that by doing an NFS
export. I don't suppose you know if CEA's "ganesha" userspace NFS
server has access to the opencache? It can cache data to disk too
which is also good for WAN applications.
>> If I just stat all the files without doing a recursive scan then it gets
>> everything from the client cache as expected without the MDS chatter - e.g.
>>
>> find /mnt/lustre -type f > /tmp/files.txt
>> cat /tmp/files.txt | xargs ls -l
>>
>> Is there any way to improve the browsing speed and cache directory
>> contents - especially for the case where I only have a single client
>> accessing an entire tree? As an aside I also noticed that a "ls -l"
>> does a getxattr - does that get cached by the client too? I can
>> imagine it might cause quite a bit of MDS chatter.
>
> So far, Lustre doesn't cache any xattr on the client beyond the file layout
> ("lustre.lov" xattr), but it is something I've been thinking about. The
> security.capability attribute is special-cased in the 1.8.4 client to not
> return any data, and beyond that there aren't any attributes that I'm aware
> of that are widely used, so I don't think there is a pressing demand for
> this, but if a case can be made for this we'll definitely look at it more
> seriously.
Yea I doubt it makes much difference I just noticed that "ls -l" did
it and was wondering what Lustre made of it.
Thanks for the insightful reply as usual Andreas,
Daire
_______________________________________________
Lustre-discuss mailing list
[email protected]
http://lists.lustre.org/mailman/listinfo/lustre-discuss
