Hi Andreas,
AWS Lustre takes a snapshot of S3 file metadata on creation.
However, I am trying to include files that are added to the S3 bucket
after Lustre filesystem creation. Since I know which files were added, I
wanted to simply inform the Lustre HSM layer of their existence. At th
On Jan 8, 2020, at 06:13, Matt Rásó-Barnett
mailto:m...@rasobarnett.com>> wrote:
Hi Kris,
I assume you are using Amazon's FSx for Lustre product
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling Lustre HSM
to S3 on AWS yourself?
I'm afraid I don't know any more than you do on
Thanks very much for the replies.. We are utilizing Lustre as a fast cache
on-top of S3.
AWS support has simply said they are not sure, but it is hard to break
through their defensive first line of support.
You have given me several avenues to explore. I did try the --exist flag,
but did not have
Hi Kris,
As people said each HSM has its own copytool with its own constraints. To work
properly with S3, additional metadata is stored in Lustre when files are
imported from a S3 bucket.
Questions specific to Amazon FSx For Lustre would be better asked to AWS
support.
Aurélien
Le 08/01/202
Hi everyone,
I don't know much about Amazon's FSx for Lustre, nor about any S3
copytool. But I do know that lhsmtool_posix (the POSIX copytool shipped
with the Lustre sources), supports an "import" operation comparable to
the "lfs hsm_register" Kris describes.
Actually, Amazon's FSx for Lust
Hi there,
we use a similar approach with the TSM Lustre copytool.
First the data is archived to a TSM Server, then one can do the
following:
$ lfs hsm_set --exists --archived --archive-id
$ lfs hsm_release
The file now exists as a released file in Lustre and is
seamlessly retrieved when the
Hi Kris,
I assume you are using Amazon's FSx for Lustre product
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling
Lustre HSM to S3 on AWS yourself?
I'm afraid I don't know any more than you do on this product but it's
something I've been keen to play with as it sounds really
We have Lustre <- HSM -> S3
We have direct modifications to S3 that occur after the Lustre filesystem
is created
I was wondering if there is any way to register a new/deleted file at the
Lustre level using HSM or other commands
Say a user uploads a file to S3, and I know the mapped path in Lust
Can you provide an example of what you're attempting to accomplish? Am I
understanding correctly, that you've got a lustre file system, you're then
writing data into this file system?
On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval wrote:
> We are using Lustre on AWS backed by S3 buckets.
>
We are using Lustre on AWS backed by S3 buckets.
When creating a new Lustre filesystem, S3 metadata can be automatically
imported into Lustre. When changes occur to the underlying S3 store,
these changes are not automatically reflected.
Is it possible to indicate the creation / deletion of the u
10 matches
Mail list logo
