Hi Kris,
I assume you are using Amazon's FSx for Lustre product
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling
Lustre HSM to S3 on AWS yourself?
I'm afraid I don't know any more than you do on this product but it's
something I've been keen to play with as it sounds really interesting.
However the process or policy-engine by which Amazon are mapping S3
objects to files in Lustre is not part of Lustre itself so you'd need
input from Amazon on this - maybe Aurelien (in BCC) can comment here?
From their overview documenation my understanding is that the Lustre FS
is meant to be short lived in this model, so you create it populated
with the contents of the bucket *at that time*, run your workload, then
archive results back and delete the FS.
So if your bucket has been updated post Lustre FS creation, you may have
to destroy the Lustre FS and recreate it to notice the changes. HSM
isn't designed to be a mechanism for synchronising two endpoints that
change independent of the other, so any change to the backend not via
the filesystem will not be tracked.
This is just me guessing without having used it however, I'll be
interested to hear if you learn more about this from Amazon.
Kind regards,
Matt
On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
We have Lustre <- HSM -> S3
We have direct modifications to S3 that occur after the Lustre filesystem
is created
I was wondering if there is any way to register a new/deleted file at the
Lustre level using HSM or other commands
Say a user uploads a file to S3, and I know the mapped path in Lustre,
I would like to do
lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
Lustre
lfs hsm_restore /path/to/file/in/S3 # Fetch file from S3 into Lustre
Thx
On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <[email protected]> wrote:
Can you provide an example of what you're attempting to accomplish? Am I
understanding correctly, that you've got a lustre file system, you're then
writing data into this file system?
On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <[email protected]> wrote:
We are using Lustre on AWS backed by S3 buckets.
When creating a new Lustre filesystem, S3 metadata can be automatically
imported into Lustre. When changes occur to the underlying S3 store,
these changes are not automatically reflected.
Is it possible to indicate the creation / deletion of the underlying S3
files after filesystem creation using HSM?
Is it possible to reimport the underlying metadata after creation?
Any pointers appreciated.
Thanks,
Kris
--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
