Hi there, we use a similar approach with the TSM Lustre copytool. First the data is archived to a TSM Server, then one can do the following:
$ lfs hsm_set --exists --archived --archive-id <archive_id> <filename> $ lfs hsm_release <filename> The file <filename> now exists as a released file in Lustre and is seamlessly retrieved when the bulk data is accessed on Lustre. Note, the the copytool "knows" the mapped path, in our TSM case (TSM object name -> Lustre filename and vice versa) e.g.: /fs/hl/ll -> /lustre/hl/ll Here is e.g. a script to sync TSM object data to a Lustre filesystem via the copytool: https://raw.githubusercontent.com/tstibor/ltsm/master/script/ltsmsync.sh Cheers Thomas On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote: > We have Lustre <- HSM -> S3 > > We have direct modifications to S3 that occur after the Lustre filesystem > is created > I was wondering if there is any way to register a new/deleted file at the > Lustre level using HSM or other commands > > Say a user uploads a file to S3, and I know the mapped path in Lustre, > I would like to do > lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in > Lustre > lfs hsm_restore /path/to/file/in/S3 # Fetch file from S3 into Lustre > > Thx > > > > > > > On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <[email protected]> wrote: > > > Can you provide an example of what you're attempting to accomplish? Am I > > understanding correctly, that you've got a lustre file system, you're then > > writing data into this file system? > > > > On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <[email protected]> wrote: > > > >> We are using Lustre on AWS backed by S3 buckets. > >> When creating a new Lustre filesystem, S3 metadata can be automatically > >> imported into Lustre. When changes occur to the underlying S3 store, > >> these changes are not automatically reflected. > >> > >> Is it possible to indicate the creation / deletion of the underlying S3 > >> files after filesystem creation using HSM? > >> Is it possible to reimport the underlying metadata after creation? > >> > >> Any pointers appreciated. > >> > >> Thanks, > >> Kris > >> > >> -- > >> Kris Kvilekval, Ph.D. > >> ViQi Inc > >> (805)-699-6081 > >> _______________________________________________ > >> lustre-discuss mailing list > >> [email protected] > >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > >> > > > > -- > Kris Kvilekval, Ph.D. > ViQi Inc > (805)-699-6081 > _______________________________________________ > lustre-discuss mailing list > [email protected] > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
