Sharding can also work, if you have a storage-connected compute grid in your environment: If you enumerate all of the directories, then use a non-recursive rsync for each one, you may be able to parallelize the workload by using several clients simultaneously. It may still max out the links of these clients (assuming your source read throughput and target write throughput bottlenecks aren't encountered first) but it may run that way for 1/100th of the time if you can use 100+ machines.
-Paul -----Original Message----- From: [email protected] <[email protected]> On Behalf Of Buterbaugh, Kevin L Sent: Thursday, October 18, 2018 2:26 PM To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] Best way to migrate data Hi Dwayne, I’m assuming you can’t just let an rsync run, possibly throttled in some way? If not, and if you’re just tapping out your network, then would it be possible to go old school? We have parts of the Medical Center here where their network connections are … um, less than robust. So they tar stuff up to a portable HD, sneaker net it to us, and we untar is from an NSD server. HTH, and I really hope that someone has a better idea than that! Kevin > On Oct 18, 2018, at 12:19 PM, [email protected] wrote: > > Hi, > > Just wondering what the best recipe for migrating a user’s home directory > content from one GFPS file system to another which hosts a larger research > GPFS file system? I’m currently using rsync and it has maxed out the client > system’s IB interface. > > Best, > Dwayne > — > Dwayne Hart | Systems Administrator IV > > CHIA, Faculty of Medicine > Memorial University of Newfoundland > 300 Prince Philip Drive > St. John’s, Newfoundland | A1B 3V6 > Craig L Dobbin Building | 4M409 > T 709 864 6631 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ccca728d2d61f4be06bcd08d6351f3650%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636754805507359478&sdata=2YAiqgqKl4CerlyCn3vJ9v9u%2FrGzbfa7aKxJ0PYV%2Fhc%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
