Hi, JC,
CFS and CEA have started working on the Lustre HSM detailed design and
implementation.
Thanks for your info on the status.
Directory migration is not supported but we will have an option to
migrate files attibutes (striping, pathname when file is migrated, ....)
in the backstore system.
So if you loose all the Lustre namespace you will be able to re-create
the Lustre namespace from the backstore but not the empty directories
nor the owner/rights of directories, files. These informations could be
backup with a separate backup tool (optimized for Lustre).
We will have a whole file pre-migration and later, when free space is
needed, we will purge the file but keep some data on the OST to avoid
staging if, for example, someone access few bytes at the beginning of
the file.
Staging will be trigered by the open or by the first read/write (this
will be a policy parameter).
Yes, it makes sense to keep the first KB also. I forgot to note that part in
my graph.
I notice there is a link below for hosting HSM related discussion. But not
much content yet.
# http://arch.lustre.org/index.php?title=Feature_HSM
Is there a way to get more updated info on current thoughts or design
considerations, or even some initial template interfaces? Just hope to avoid
too distant from what you people are thinking or actually working on.
Thanks,
Weikuan
JC
Weikuan Yu wrote:
Resent with an account on the list. Sorry for repetitions.
Weikuan
Hi,
It has been sometime since last discussion on Lustre HSM plan. Folks
here at ORNL are interested to get this work going. Attached is a
diagram about our initial idea and plan for stage 1. What shown is
only about staging. But purging/migration should be more or less the
same path from step 2 to step 9. Only difference is that they may not
be in the critical path of client IO. As you may realize, this is
just to show the idea for stage 1 of the Lustre HSM plan.
Along with this diagram, I have also a couple of doubts.
1) I know a point was made not to archive striping info. Is Lustre
HSM going to support migration of directories, or just objects of
file data? If yes, does it imply that we need to archive striping
info too?
2) I know the earlier CMOBD is supposed to situated entirely under
OST/MDT. That makes sense for stage 2, i.e. partial file or
per-extent migration. Would it just sufficient to have a caching
module (HSC) under MDS to do per-file migration for stage 1? This
also means that we do not have to save the per-extent migration
status in the metadata, which, if implemented, would just fight for
the limited space with the stripe map descriptors.
3) I put an HSS there as something similar to the current OST, meant
to serve multiple HSOBD in the hope that it will provide flexibility.
Please comment on the questions and diagram whether they make sense
or not.
We are also curious to know if CFS and CEA have started in this
direction. If so, how could we join hands in some way.
Thanks,
Weikuan
------------------------------------------------------------------------
------------------------------------------------------------------------
_______________________________________________
Lustre-discuss mailing list
[email protected]
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
_______________________________________________
Lustre-discuss mailing list
[email protected]
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
