Re: Need to copy a 200GB directory
OK, my apologies for hijacking this thread, I haven't been on a mailing list in forever but I will apply proper etiquette. Can I just ask what you mean by "top post" though? On Tue, Jun 27, 2017 at 9:25 PM Greg Rundlett (freephile) < g...@freephile.com> wrote: > Hi Anthony! Welcome! > > You can just reply to the list in general, but it doesn't hurt to reply-all > > You should always start a new topic with a new thread ;-). And never top > post (unless you're me and using a phone) > > ~ Greg > > On Jun 27, 2017 8:00 PM, "R. Anthony Lomartire"> wrote: > >> Also sorry idk if there is an intro thread or anything, but I've been a >> lurker for a while this has been my first actual post I think. I don't know >> if I should reply all or just send my reply to the GNHLUG email address? >> >> Anyways just quickly, I'm Tony and I'm in ad tech. We use machine >> learning to help advertisers optimize their ROI. At first I thought it >> would be lame, but at least it was a job, but gradually I have become more >> and more interested in ad tech and it is actually kinda cool. Ok so hiii! >> >> On Tue, Jun 27, 2017 at 7:55 PM R. Anthony Lomartire < >> opensourcek...@gmail.com> wrote: >> >>> No offense or anything but I find it amusing that one of the most active >>> threads on this mailer has been about copying a bit of data :D >>> >>> On Tue, Jun 27, 2017 at 6:29 PM Matt Minuti >>> wrote: >>> My muscle memory always puts the flags "-avz" (sometimes I even remember to add a P in there), so there must have been one point in time where you had to specify compression. Might still be the case. On Tue, Jun 27, 2017, 12:02 PM mark wrote: > My mistake. I wrote encryption when I meant compression, as I belive > rsync always compresses--but I could be mistaken about that, too! > > Mark > On Jun 27, 2017 11:55 AM, "Tom Buskey" wrote: > >> rsync doesn't encrypt if there's no remote, as in this case. >> >> To be pedantic, rsync to remotes uses ssh by default but it can use >> rsh which has no encryption. Some older versions of SSH allowed you to >> specify the encryption. I recall using XOR encryption for faster >> operation >> where security was not needed. >> >> Encryption typically does some compression. If you compress 2x, >> you're doubling the bits through the pipe in the same time. If the >> encryption/compression computation at either end is faster than than the >> uncompressed bandwidth, you'll have faster throughput. That's very >> typical >> on newer multicore, high GHz CPUs. >> >> On Mon, Jun 26, 2017 at 4:11 PM, mark wrote: >> >>> Locally, cp is faster because you cannot make rsync not encrypt, but >>> the restart-from-where-it-stopped feature of rsync makes it worth the >>> wait. >>> >>> Mark >>> On Jun 26, 2017 3:18 PM, "Charles Farinella" < >>> cfarine...@appropriatesolutions.com> wrote: >>> We need to copy a large (200+GB) directory from one filesystem to another, both locally mounted. I'm unsure as to what I should use to do this, cp, rsync, dd? Any suggestions appreciated. Thanks. --charlie Charlie Farinella Systems Administrator Appropriate Solutions, Inc. 1-603-924-6079 <(603)%20924-6079> ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> ___ >>> gnhlug-discuss mailing list >>> gnhlug-discuss@mail.gnhlug.org >>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >>> >> ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> >> ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
Hi Anthony! Welcome! You can just reply to the list in general, but it doesn't hurt to reply-all You should always start a new topic with a new thread ;-). And never top post (unless you're me and using a phone) ~ Greg On Jun 27, 2017 8:00 PM, "R. Anthony Lomartire"wrote: > Also sorry idk if there is an intro thread or anything, but I've been a > lurker for a while this has been my first actual post I think. I don't know > if I should reply all or just send my reply to the GNHLUG email address? > > Anyways just quickly, I'm Tony and I'm in ad tech. We use machine learning > to help advertisers optimize their ROI. At first I thought it would be > lame, but at least it was a job, but gradually I have become more and more > interested in ad tech and it is actually kinda cool. Ok so hiii! > > On Tue, Jun 27, 2017 at 7:55 PM R. Anthony Lomartire < > opensourcek...@gmail.com> wrote: > >> No offense or anything but I find it amusing that one of the most active >> threads on this mailer has been about copying a bit of data :D >> >> On Tue, Jun 27, 2017 at 6:29 PM Matt Minuti >> wrote: >> >>> My muscle memory always puts the flags "-avz" (sometimes I even remember >>> to add a P in there), so there must have been one point in time where you >>> had to specify compression. Might still be the case. >>> >>> On Tue, Jun 27, 2017, 12:02 PM mark wrote: >>> My mistake. I wrote encryption when I meant compression, as I belive rsync always compresses--but I could be mistaken about that, too! Mark On Jun 27, 2017 11:55 AM, "Tom Buskey" wrote: > rsync doesn't encrypt if there's no remote, as in this case. > > To be pedantic, rsync to remotes uses ssh by default but it can use > rsh which has no encryption. Some older versions of SSH allowed you to > specify the encryption. I recall using XOR encryption for faster > operation > where security was not needed. > > Encryption typically does some compression. If you compress 2x, > you're doubling the bits through the pipe in the same time. If the > encryption/compression computation at either end is faster than than the > uncompressed bandwidth, you'll have faster throughput. That's very > typical > on newer multicore, high GHz CPUs. > > On Mon, Jun 26, 2017 at 4:11 PM, mark wrote: > >> Locally, cp is faster because you cannot make rsync not encrypt, but >> the restart-from-where-it-stopped feature of rsync makes it worth the >> wait. >> >> Mark >> On Jun 26, 2017 3:18 PM, "Charles Farinella" > appropriatesolutions.com> wrote: >> >>> We need to copy a large (200+GB) directory from one filesystem to >>> another, both locally mounted. >>> >>> I'm unsure as to what I should use to do this, cp, rsync, dd? >>> >>> Any suggestions appreciated. >>> >>> Thanks. >>> >>> --charlie >>> >>> Charlie Farinella >>> Systems Administrator >>> Appropriate Solutions, Inc. >>> 1-603-924-6079 <(603)%20924-6079> >>> >>> ___ >>> gnhlug-discuss mailing list >>> gnhlug-discuss@mail.gnhlug.org >>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >>> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> >> > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> ___ >>> gnhlug-discuss mailing list >>> gnhlug-discuss@mail.gnhlug.org >>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >> > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
Also sorry idk if there is an intro thread or anything, but I've been a lurker for a while this has been my first actual post I think. I don't know if I should reply all or just send my reply to the GNHLUG email address? Anyways just quickly, I'm Tony and I'm in ad tech. We use machine learning to help advertisers optimize their ROI. At first I thought it would be lame, but at least it was a job, but gradually I have become more and more interested in ad tech and it is actually kinda cool. Ok so hiii! On Tue, Jun 27, 2017 at 7:55 PM R. Anthony Lomartire < opensourcek...@gmail.com> wrote: > No offense or anything but I find it amusing that one of the most active > threads on this mailer has been about copying a bit of data :D > > On Tue, Jun 27, 2017 at 6:29 PM Matt Minutiwrote: > >> My muscle memory always puts the flags "-avz" (sometimes I even remember >> to add a P in there), so there must have been one point in time where you >> had to specify compression. Might still be the case. >> >> On Tue, Jun 27, 2017, 12:02 PM mark wrote: >> >>> My mistake. I wrote encryption when I meant compression, as I belive >>> rsync always compresses--but I could be mistaken about that, too! >>> >>> Mark >>> On Jun 27, 2017 11:55 AM, "Tom Buskey" wrote: >>> rsync doesn't encrypt if there's no remote, as in this case. To be pedantic, rsync to remotes uses ssh by default but it can use rsh which has no encryption. Some older versions of SSH allowed you to specify the encryption. I recall using XOR encryption for faster operation where security was not needed. Encryption typically does some compression. If you compress 2x, you're doubling the bits through the pipe in the same time. If the encryption/compression computation at either end is faster than than the uncompressed bandwidth, you'll have faster throughput. That's very typical on newer multicore, high GHz CPUs. On Mon, Jun 26, 2017 at 4:11 PM, mark wrote: > Locally, cp is faster because you cannot make rsync not encrypt, but > the restart-from-where-it-stopped feature of rsync makes it worth the > wait. > > Mark > On Jun 26, 2017 3:18 PM, "Charles Farinella" < > cfarine...@appropriatesolutions.com> wrote: > >> We need to copy a large (200+GB) directory from one filesystem to >> another, both locally mounted. >> >> I'm unsure as to what I should use to do this, cp, rsync, dd? >> >> Any suggestions appreciated. >> >> Thanks. >> >> --charlie >> >> Charlie Farinella >> Systems Administrator >> Appropriate Solutions, Inc. >> 1-603-924-6079 <(603)%20924-6079> >> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> >> > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > > ___ >>> gnhlug-discuss mailing list >>> gnhlug-discuss@mail.gnhlug.org >>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
No offense or anything but I find it amusing that one of the most active threads on this mailer has been about copying a bit of data :D On Tue, Jun 27, 2017 at 6:29 PM Matt Minutiwrote: > My muscle memory always puts the flags "-avz" (sometimes I even remember > to add a P in there), so there must have been one point in time where you > had to specify compression. Might still be the case. > > On Tue, Jun 27, 2017, 12:02 PM mark wrote: > >> My mistake. I wrote encryption when I meant compression, as I belive >> rsync always compresses--but I could be mistaken about that, too! >> >> Mark >> On Jun 27, 2017 11:55 AM, "Tom Buskey" wrote: >> >>> rsync doesn't encrypt if there's no remote, as in this case. >>> >>> To be pedantic, rsync to remotes uses ssh by default but it can use rsh >>> which has no encryption. Some older versions of SSH allowed you to specify >>> the encryption. I recall using XOR encryption for faster operation where >>> security was not needed. >>> >>> Encryption typically does some compression. If you compress 2x, you're >>> doubling the bits through the pipe in the same time. If the >>> encryption/compression computation at either end is faster than than the >>> uncompressed bandwidth, you'll have faster throughput. That's very typical >>> on newer multicore, high GHz CPUs. >>> >>> On Mon, Jun 26, 2017 at 4:11 PM, mark wrote: >>> Locally, cp is faster because you cannot make rsync not encrypt, but the restart-from-where-it-stopped feature of rsync makes it worth the wait. Mark On Jun 26, 2017 3:18 PM, "Charles Farinella" < cfarine...@appropriatesolutions.com> wrote: > We need to copy a large (200+GB) directory from one filesystem to > another, both locally mounted. > > I'm unsure as to what I should use to do this, cp, rsync, dd? > > Any suggestions appreciated. > > Thanks. > > --charlie > > Charlie Farinella > Systems Administrator > Appropriate Solutions, Inc. > 1-603-924-6079 <(603)%20924-6079> > > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
My muscle memory always puts the flags "-avz" (sometimes I even remember to add a P in there), so there must have been one point in time where you had to specify compression. Might still be the case. On Tue, Jun 27, 2017, 12:02 PM markwrote: > My mistake. I wrote encryption when I meant compression, as I belive rsync > always compresses--but I could be mistaken about that, too! > > Mark > On Jun 27, 2017 11:55 AM, "Tom Buskey" wrote: > >> rsync doesn't encrypt if there's no remote, as in this case. >> >> To be pedantic, rsync to remotes uses ssh by default but it can use rsh >> which has no encryption. Some older versions of SSH allowed you to specify >> the encryption. I recall using XOR encryption for faster operation where >> security was not needed. >> >> Encryption typically does some compression. If you compress 2x, you're >> doubling the bits through the pipe in the same time. If the >> encryption/compression computation at either end is faster than than the >> uncompressed bandwidth, you'll have faster throughput. That's very typical >> on newer multicore, high GHz CPUs. >> >> On Mon, Jun 26, 2017 at 4:11 PM, mark wrote: >> >>> Locally, cp is faster because you cannot make rsync not encrypt, but the >>> restart-from-where-it-stopped feature of rsync makes it worth the wait. >>> >>> Mark >>> On Jun 26, 2017 3:18 PM, "Charles Farinella" < >>> cfarine...@appropriatesolutions.com> wrote: >>> We need to copy a large (200+GB) directory from one filesystem to another, both locally mounted. I'm unsure as to what I should use to do this, cp, rsync, dd? Any suggestions appreciated. Thanks. --charlie Charlie Farinella Systems Administrator Appropriate Solutions, Inc. 1-603-924-6079 <(603)%20924-6079> ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> ___ >>> gnhlug-discuss mailing list >>> gnhlug-discuss@mail.gnhlug.org >>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >>> >> ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
I typically use rsync instead of cp when I can. There can be issues (and there's too much below I know): lots of small files can slow things down because of the comparision (checksum, dates, names). zfs send/receive uses a different method for remote replication that avoids this. If you're on cygwin, there are restrictions (I can't recall specifics) compared to Unixen. Different file systems may act different. ACLs are not copied. Other permissions might not work too (NTFS to FAT and it'll recopy instead of skip) MacOSX HFS files have 2 forks, resource (with icons, etc) and data. On older versions (panther?), cp would copy both forks. Rsync would only deal with the data fork. NTFS has forks IIRC but I think cygwin can deal. It's no fun having the user complain all the icons on the MacOS9 desktop are wrong because the move/restore used rsync. On Tue, Jun 27, 2017 at 11:55 AM, Tom Buskeywrote: > rsync doesn't encrypt if there's no remote, as in this case. > > To be pedantic, rsync to remotes uses ssh by default but it can use rsh > which has no encryption. Some older versions of SSH allowed you to specify > the encryption. I recall using XOR encryption for faster operation where > security was not needed. > > Encryption typically does some compression. If you compress 2x, you're > doubling the bits through the pipe in the same time. If the > encryption/compression computation at either end is faster than than the > uncompressed bandwidth, you'll have faster throughput. That's very typical > on newer multicore, high GHz CPUs. > > On Mon, Jun 26, 2017 at 4:11 PM, mark wrote: > >> Locally, cp is faster because you cannot make rsync not encrypt, but the >> restart-from-where-it-stopped feature of rsync makes it worth the wait. >> >> Mark >> On Jun 26, 2017 3:18 PM, "Charles Farinella" < >> cfarine...@appropriatesolutions.com> wrote: >> >>> We need to copy a large (200+GB) directory from one filesystem to >>> another, both locally mounted. >>> >>> I'm unsure as to what I should use to do this, cp, rsync, dd? >>> >>> Any suggestions appreciated. >>> >>> Thanks. >>> >>> --charlie >>> >>> Charlie Farinella >>> Systems Administrator >>> Appropriate Solutions, Inc. >>> 1-603-924-6079 <(603)%20924-6079> >>> >>> ___ >>> gnhlug-discuss mailing list >>> gnhlug-discuss@mail.gnhlug.org >>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >>> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> >> > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
My mistake. I wrote encryption when I meant compression, as I belive rsync always compresses--but I could be mistaken about that, too! Mark On Jun 27, 2017 11:55 AM, "Tom Buskey"wrote: > rsync doesn't encrypt if there's no remote, as in this case. > > To be pedantic, rsync to remotes uses ssh by default but it can use rsh > which has no encryption. Some older versions of SSH allowed you to specify > the encryption. I recall using XOR encryption for faster operation where > security was not needed. > > Encryption typically does some compression. If you compress 2x, you're > doubling the bits through the pipe in the same time. If the > encryption/compression computation at either end is faster than than the > uncompressed bandwidth, you'll have faster throughput. That's very typical > on newer multicore, high GHz CPUs. > > On Mon, Jun 26, 2017 at 4:11 PM, mark wrote: > >> Locally, cp is faster because you cannot make rsync not encrypt, but the >> restart-from-where-it-stopped feature of rsync makes it worth the wait. >> >> Mark >> On Jun 26, 2017 3:18 PM, "Charles Farinella" < >> cfarine...@appropriatesolutions.com> wrote: >> >>> We need to copy a large (200+GB) directory from one filesystem to >>> another, both locally mounted. >>> >>> I'm unsure as to what I should use to do this, cp, rsync, dd? >>> >>> Any suggestions appreciated. >>> >>> Thanks. >>> >>> --charlie >>> >>> Charlie Farinella >>> Systems Administrator >>> Appropriate Solutions, Inc. >>> 1-603-924-6079 <(603)%20924-6079> >>> >>> ___ >>> gnhlug-discuss mailing list >>> gnhlug-discuss@mail.gnhlug.org >>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >>> >>> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> >> > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
rsync doesn't encrypt if there's no remote, as in this case. To be pedantic, rsync to remotes uses ssh by default but it can use rsh which has no encryption. Some older versions of SSH allowed you to specify the encryption. I recall using XOR encryption for faster operation where security was not needed. Encryption typically does some compression. If you compress 2x, you're doubling the bits through the pipe in the same time. If the encryption/compression computation at either end is faster than than the uncompressed bandwidth, you'll have faster throughput. That's very typical on newer multicore, high GHz CPUs. On Mon, Jun 26, 2017 at 4:11 PM, markwrote: > Locally, cp is faster because you cannot make rsync not encrypt, but the > restart-from-where-it-stopped feature of rsync makes it worth the wait. > > Mark > On Jun 26, 2017 3:18 PM, "Charles Farinella" appropriatesolutions.com> wrote: > >> We need to copy a large (200+GB) directory from one filesystem to >> another, both locally mounted. >> >> I'm unsure as to what I should use to do this, cp, rsync, dd? >> >> Any suggestions appreciated. >> >> Thanks. >> >> --charlie >> >> Charlie Farinella >> Systems Administrator >> Appropriate Solutions, Inc. >> 1-603-924-6079 <(603)%20924-6079> >> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> >> > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
rsync it is, thanks to all. --charlie On Mon, Jun 26, 2017 at 4:11 PM, markwrote: > Locally, cp is faster because you cannot make rsync not encrypt, but the > restart-from-where-it-stopped feature of rsync makes it worth the wait. > > Mark > > On Jun 26, 2017 3:18 PM, "Charles Farinella" > wrote: >> >> We need to copy a large (200+GB) directory from one filesystem to another, >> both locally mounted. >> >> I'm unsure as to what I should use to do this, cp, rsync, dd? >> >> Any suggestions appreciated. >> >> Thanks. >> >> --charlie >> >> Charlie Farinella >> Systems Administrator >> Appropriate Solutions, Inc. >> 1-603-924-6079 >> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> > > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > -- Charlie Farinella 14 East Ridge Drive Peterborough, NH 03458 603-924-1977: Home 603-785-3320: Mobile ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
Locally, cp is faster because you cannot make rsync not encrypt, but the restart-from-where-it-stopped feature of rsync makes it worth the wait. Mark On Jun 26, 2017 3:18 PM, "Charles Farinella" < cfarine...@appropriatesolutions.com> wrote: > We need to copy a large (200+GB) directory from one filesystem to another, > both locally mounted. > > I'm unsure as to what I should use to do this, cp, rsync, dd? > > Any suggestions appreciated. > > Thanks. > > --charlie > > Charlie Farinella > Systems Administrator > Appropriate Solutions, Inc. > 1-603-924-6079 > > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
rsync -van --stats --exclude images/ --exclude other_big_dir/ /var/source/ /var/destination/ -v verbose -a archive mode (preserves perms., recursive, etc.) -n dry-run Make sure to use trailing slashes if transferring directories Use excludes to get it working before you move the biggest directories; or just do a small non-recursive rsync first. Greg Rundlett https://eQuality-Tech.com https://freephile.org On Mon, Jun 26, 2017 at 3:29 PM, David Rosewrote: > rsync -a source destination is all it should take. > > On Jun 26, 2017 3:25 PM, "Dan Garthwaite" wrote: > >> Ditto Ken on two points: 200GB isn't that large (I've worked in an >> animation studio) and rsync is restartable. I'd go with rsync. >> >> It has a dizzying array of options and even more finer points. You don't >> need the rsync daemon. Try to use full paths. Include trailing slashes if >> copying directories. Experiment and then write a bash script. >> >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> >> > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
Yeah, dd is wrong, I'll play with cp and rsync and pick one. Thanks. --charlie Charlie Farinella Systems Administrator Appropriate Solutions, Inc. 1-603-924-6079 On Mon, Jun 26, 2017 at 3:18 PM, Ken D'Ambrosiowrote: > 200 GB on locally mounted filesystems just isn't all *THAT* much. I'm not > quite sure how you'd use 'dd', but cp or rsync should do the trick just > fine. Note that rsync has the added benefit of being able to, essentially, > start from where you failed -- but I usually reserve that for network file > transfers that take long enough I'm worried a brownout or something might > interrupt it. > > $.02, > > -Ken > > > > On 2017-06-26 15:11, Charles Farinella wrote: > >> We need to copy a large (200+GB) directory from one filesystem to >> another, both locally mounted. >> >> I'm unsure as to what I should use to do this, cp, rsync, dd? >> >> Any suggestions appreciated. >> >> Thanks. >> >> --charlie >> >> Charlie Farinella >> Systems Administrator >> Appropriate Solutions, Inc. >> 1-603-924-6079 >> ___ >> gnhlug-discuss mailing list >> gnhlug-discuss@mail.gnhlug.org >> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ >> > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
rsync -a source destination is all it should take. On Jun 26, 2017 3:25 PM, "Dan Garthwaite"wrote: > Ditto Ken on two points: 200GB isn't that large (I've worked in an > animation studio) and rsync is restartable. I'd go with rsync. > > It has a dizzying array of options and even more finer points. You don't > need the rsync daemon. Try to use full paths. Include trailing slashes if > copying directories. Experiment and then write a bash script. > > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ > > ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
I definitely would use rsync. I regularly copy 2-3 TB from between filesystems, both locally and over networks, using it. cp would work, but rsync is better if you do need to interrupt and restart or if you just need to synchronize changes. On 6/26/2017 3:11 PM, Charles Farinella wrote: > We need to copy a large (200+GB) directory from one filesystem to > another, both locally mounted. > > I'm unsure as to what I should use to do this, cp, rsync, dd? > > Any suggestions appreciated. > > Thanks. > > --charlie > > Charlie Farinella > Systems Administrator > Appropriate Solutions, Inc. > 1-603-924-6079 > > > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
Ditto Ken on two points: 200GB isn't that large (I've worked in an animation studio) and rsync is restartable. I'd go with rsync. It has a dizzying array of options and even more finer points. You don't need the rsync daemon. Try to use full paths. Include trailing slashes if copying directories. Experiment and then write a bash script. ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Need to copy a 200GB directory
200 GB on locally mounted filesystems just isn't all *THAT* much. I'm not quite sure how you'd use 'dd', but cp or rsync should do the trick just fine. Note that rsync has the added benefit of being able to, essentially, start from where you failed -- but I usually reserve that for network file transfers that take long enough I'm worried a brownout or something might interrupt it. $.02, -Ken On 2017-06-26 15:11, Charles Farinella wrote: > We need to copy a large (200+GB) directory from one filesystem to > another, both locally mounted. > > I'm unsure as to what I should use to do this, cp, rsync, dd? > > Any suggestions appreciated. > > Thanks. > > --charlie > > Charlie Farinella > Systems Administrator > Appropriate Solutions, Inc. > 1-603-924-6079 > ___ > gnhlug-discuss mailing list > gnhlug-discuss@mail.gnhlug.org > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/