Hi So when we have migrated 1.6PB of data from one GPFS filesystems to another GPFS (over IB), we used dcp in github (with mmdsh). It just can be problematic to compile.
I have used rsync with attrib and ACLs’s preserved in my previous job – aka rsync -aAvz But DCP parallelises better, checksumming files and dirs. works and we used that to ensure nothing was lost. Worth a go! Regards, Chris Schlipalius Team Lead, Data Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO) 13 Burvill Court Kensington WA 6151 Australia Tel +61 8 6436 8815 Email [email protected] Web www.pawsey.org.au <http://www.pawsey.org.au/> On 23/10/18, 4:08 am, "[email protected] on behalf of [email protected]" <[email protected] on behalf of [email protected]> wrote: Send gpfsug-discuss mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Best way to migrate data (Ryan Novosielski) 2. Re: Best way to migrate data (Sven Oehme) 3. Re: Best way to migrate data : mmfind ... mmxcp (Marc A Kaplan) ---------------------------------------------------------------------- Message: 1 Date: Mon, 22 Oct 2018 15:21:06 +0000 From: Ryan Novosielski <[email protected]> To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] Best way to migrate data Message-ID: <[email protected]> Content-Type: text/plain; charset="utf-8" It seems like the primary way that this helps us is that we transfer user home directories and many of them have VERY large numbers of small files (in the millions), so running multiple simultaneous rsyncs allows the transfer to continue past that one slow area. I guess it balances the bandwidth constraint and the I/O constraints on generating a file list. There are unfortunately one or two known bugs that slow it down ? it keeps track of its rsync PIDs but sometimes a former rsync PID is reused by the system which it counts against the number of running rsyncs. It can also think rsync is still running at the end when it?s really something else now using the PID. I know the author is looking at that. For shorter transfers, you likely won?t run into this. I?m not sure I have the time or the programming ability to make this happen, but it seems to me that one could make some major gains by replacing fpart with mmfind in a GPFS environment. Generating lists of files takes a significant amount of time and mmfind can probably do it faster than anything else that does not have direct access to GPFS metadata. > On Oct 19, 2018, at 6:37 AM, [email protected] wrote: > > Thank you Ryan. I?ll have a more in-depth look at this application later today and see how it deals with some of the large genetic files that are generated by the sequencer. By copying it from GPFS fs to another GPFS fs. > > Best, > Dwayne > ? > Dwayne Hart | Systems Administrator IV > > CHIA, Faculty of Medicine > Memorial University of Newfoundland > 300 Prince Philip Drive > St. John?s, Newfoundland | A1B 3V6 > Craig L Dobbin Building | 4M409 > T 709 864 6631 > >> On Oct 19, 2018, at 7:04 AM, Ryan Novosielski <[email protected]> wrote: >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> We use parsyncfp. Our target is not GPFS, though. I was really hoping >> to hear about something snazzier for GPFS-GPFS. Lenovo would probably >> tell you that HSM is the way to go (we asked something similar for a >> replacement for our current setup or for distributed storage). >> >>> On 10/18/2018 01:19 PM, [email protected] wrote: >>> Hi, >>> >>> Just wondering what the best recipe for migrating a user?s home >>> directory content from one GFPS file system to another which hosts >>> a larger research GPFS file system? I?m currently using rsync and >>> it has maxed out the client system?s IB interface. >>> >>> Best, Dwayne ? Dwayne Hart | Systems Administrator IV >>> >>> CHIA, Faculty of Medicine Memorial University of Newfoundland 300 >>> Prince Philip Drive St. John?s, Newfoundland | A1B 3V6 Craig L >>> Dobbin Building | 4M409 T 709 864 6631 >>> _______________________________________________ gpfsug-discuss >>> mailing list gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> - -- >> ____ >> || \\UTGERS, |----------------------*O*------------------------ >> ||_// the State | Ryan Novosielski - [email protected] >> || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus >> || \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark >> `' >> -----BEGIN PGP SIGNATURE----- >> >> iEYEARECAAYFAlvI51AACgkQmb+gadEcsb62SQCfWBAru3KkJd+UftG2BXaRzjTG >> p/wAn0mpC5XCZc50fZfMPRRXR40HsmEk >> =dMDg >> -----END PGP SIGNATURE----- >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ------------------------------ Message: 2 Date: Mon, 22 Oct 2018 11:11:06 -0700 From: Sven Oehme <[email protected]> To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] Best way to migrate data Message-ID: <CALssuR35D=AEchSzezNzqpB=tx2+cqx-omjl8xwardhko5r...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" i am not sure if that was mentioned already but in some version of V5.0.X based on my suggestion a tool was added by mark on a AS-IS basis (thanks mark) to do what you want with one exception : /usr/lpp/mmfs/samples/ilm/mmxcp -h Usage: /usr/lpp/mmfs/samples/ilm/mmxcp -t target -p strip_count source_pathname1 source_pathname2 ... Run "cp" in a mmfind ... -xarg ... pipeline, e.g. mmfind -polFlags '-N all -g /gpfs/tmp' /gpfs/source -gpfsWeight DIRECTORY_HASH -xargs mmxcp -t /target -p 2 Options: -t target_path : Copy files to this path. -p strip_count : Remove this many directory names from the pathnames of the source files. -a : pass -a to cp -v : pass -v to cp this is essentially a parallel copy tool using the policy with all its goddies. the one critical part thats missing is that it doesn't copy any GPFS specific metadata which unfortunate includes NFSV4 ACL's. the reason for that is that GPFS doesn't expose the NFSV4 ACl's via xattrs nor does any of the regular Linux tools uses the proprietary interface into GPFS to extract and apply them (this is what allows this magic unsupported version of rsync https://github.com/gpfsug/gpfsug-tools/tree/master/bin/rsync to transfer the acls and other attributes). so a worth while RFE would be to either expose all special GPFS bits as xattrs or provide at least a maintained version of sync, cp or whatever which allows the transfer of this data. Sven On Mon, Oct 22, 2018 at 10:52 AM Ryan Novosielski <[email protected]> wrote: > It seems like the primary way that this helps us is that we transfer user > home directories and many of them have VERY large numbers of small files > (in the millions), so running multiple simultaneous rsyncs allows the > transfer to continue past that one slow area. I guess it balances the > bandwidth constraint and the I/O constraints on generating a file list. > There are unfortunately one or two known bugs that slow it down ? it keeps > track of its rsync PIDs but sometimes a former rsync PID is reused by the > system which it counts against the number of running rsyncs. It can also > think rsync is still running at the end when it?s really something else now > using the PID. I know the author is looking at that. For shorter transfers, > you likely won?t run into this. > > I?m not sure I have the time or the programming ability to make this > happen, but it seems to me that one could make some major gains by > replacing fpart with mmfind in a GPFS environment. Generating lists of > files takes a significant amount of time and mmfind can probably do it > faster than anything else that does not have direct access to GPFS metadata. > > > On Oct 19, 2018, at 6:37 AM, [email protected] wrote: > > > > Thank you Ryan. I?ll have a more in-depth look at this application later > today and see how it deals with some of the large genetic files that are > generated by the sequencer. By copying it from GPFS fs to another GPFS fs. > > > > Best, > > Dwayne > > ? > > Dwayne Hart | Systems Administrator IV > > > > CHIA, Faculty of Medicine > > Memorial University of Newfoundland > > 300 Prince Philip Drive > > St. John?s, Newfoundland | A1B 3V6 > > Craig L Dobbin Building | 4M409 > > T 709 864 6631 <(709)%20864-6631> > > > >> On Oct 19, 2018, at 7:04 AM, Ryan Novosielski <[email protected]> > wrote: > >> > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> We use parsyncfp. Our target is not GPFS, though. I was really hoping > >> to hear about something snazzier for GPFS-GPFS. Lenovo would probably > >> tell you that HSM is the way to go (we asked something similar for a > >> replacement for our current setup or for distributed storage). > >> > >>> On 10/18/2018 01:19 PM, [email protected] wrote: > >>> Hi, > >>> > >>> Just wondering what the best recipe for migrating a user?s home > >>> directory content from one GFPS file system to another which hosts > >>> a larger research GPFS file system? I?m currently using rsync and > >>> it has maxed out the client system?s IB interface. > >>> > >>> Best, Dwayne ? Dwayne Hart | Systems Administrator IV > >>> > >>> CHIA, Faculty of Medicine Memorial University of Newfoundland 300 > >>> Prince Philip Drive St. John?s, Newfoundland | A1B 3V6 Craig L > >>> Dobbin Building | 4M409 T 709 864 6631 <(709)%20864-6631> > >>> _______________________________________________ gpfsug-discuss > >>> mailing list gpfsug-discuss at spectrumscale.org > >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >>> > >> > >> - -- > >> ____ > >> || \\UTGERS, |----------------------*O*------------------------ > >> ||_// the State | Ryan Novosielski - [email protected] > >> || \\ University | Sr. Technologist - 973/972.0922 <(973)%20972-0922> > ~*~ RBHS Campus > >> || \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark > >> `' > >> -----BEGIN PGP SIGNATURE----- > >> > >> iEYEARECAAYFAlvI51AACgkQmb+gadEcsb62SQCfWBAru3KkJd+UftG2BXaRzjTG > >> p/wAn0mpC5XCZc50fZfMPRRXR40HsmEk > >> =dMDg > >> -----END PGP SIGNATURE----- > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20181022/5eda4214/attachment-0001.html> ------------------------------ Message: 3 Date: Mon, 22 Oct 2018 16:08:49 -0400 From: "Marc A Kaplan" <[email protected]> To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] Best way to migrate data : mmfind ... mmxcp Message-ID: <off1cd736d.7f0a9aea-on8525832e.006e23d5-8525832e.006ea...@notes.na.collabserv.com> Content-Type: text/plain; charset="us-ascii" Rather than hack rsync or cp ... I proposed a smallish utility that would copy those extended attributes and ACLs that cp -a just skips over. This can be done using the documented GPFS APIs that were designed for backup and restore of files. SMOP and then add it as an option to samples/ilm/mmxcp Sorry I haven't gotten around to doing this ... Seems like a modest sized project... Avoids boiling the ocean and reinventing or hacking rsync. -- marc K -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20181022/35e81523/attachment.html> ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 81, Issue 44 ********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
