Re: [gentoo-user] Copying data efficiently
On Saturday 17 May 2014 04:33:57 meino.cra...@gmx.de wrote: Hi, is there any tool in the Gentoo portage which may speed up (make it mopre efficient) the following task: On my HD there are data I want to copy to two identical external HDs. These HDs are of the same type/model and each is separately connectable via USB to my PC (...these two of the typical mobile external USB-HDs). Instead of copying the data twice from my PC to eah of the HDs I want to do it once...like I would be able to give the cp-command two instead of one target where to copy two. The result should be two identically populated external HDs, each of them useable/readable without the need to the other one. What tool of the portage tree I able to accomplish this task? Best regards, mcc 1. You could set up the two ext drives as a mirrored RAID and use your copying/tar-ing/dd tool of choice. 2. Or you could use pipe and tee to split the feed into any devices you want, e.g.: pv /dev/sda1 | tee (dd of=/dev/sdb1) (dd of=/dev/sdc1) 3. Or you could use a sequential copy: cp -a /home /dev/sdb1/ cp -a /home /dev/sdc1 NOTES: a) Unlike other commands, pv will give you a progress bar so that you know how long your back up is taking. b) The 3rd example above will copy sequentially, but depending on the size of the file(s) the second copy may be read from cache. c) If you're doing this over the network then you can use nc in listening mode at the receiving end, or ssh if the network is untrusted. I'm interested to see what other ways will be suggested for this task. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Copying data efficiently
Mick michaelkintz...@gmail.com [14-05-17 09:48]: On Saturday 17 May 2014 04:33:57 meino.cra...@gmx.de wrote: Hi, is there any tool in the Gentoo portage which may speed up (make it mopre efficient) the following task: On my HD there are data I want to copy to two identical external HDs. These HDs are of the same type/model and each is separately connectable via USB to my PC (...these two of the typical mobile external USB-HDs). Instead of copying the data twice from my PC to eah of the HDs I want to do it once...like I would be able to give the cp-command two instead of one target where to copy two. The result should be two identically populated external HDs, each of them useable/readable without the need to the other one. What tool of the portage tree I able to accomplish this task? Best regards, mcc 1. You could set up the two ext drives as a mirrored RAID and use your copying/tar-ing/dd tool of choice. 2. Or you could use pipe and tee to split the feed into any devices you want, e.g.: pv /dev/sda1 | tee (dd of=/dev/sdb1) (dd of=/dev/sdc1) 3. Or you could use a sequential copy: cp -a /home /dev/sdb1/ cp -a /home /dev/sdc1 NOTES: a) Unlike other commands, pv will give you a progress bar so that you know how long your back up is taking. b) The 3rd example above will copy sequentially, but depending on the size of the file(s) the second copy may be read from cache. c) If you're doing this over the network then you can use nc in listening mode at the receiving end, or ssh if the network is untrusted. I'm interested to see what other ways will be suggested for this task. -- Regards, Mick Hi Mick, thank your reply! :) From your numbering of the possibilities... 1.) ...I am no RAID guru and would try this later with data, which are not valuable... 2.) That looks interesting! Unfortunately it seems to copy device contents on low level instead of files. The source are directory structure -- not whole devices... Or did I overlook an option mentioned in the manpage...? 3.) The files I want to copy are in the size of some GB each. So the cache isnt big enough to hold ALL files for the second part. Best regards, mcc
Re: [gentoo-user] Copying data efficiently
On Sat, 17 May 2014 09:59:08 +0200, meino.cra...@gmx.de wrote: 3. Or you could use a sequential copy: cp -a /home /dev/sdb1/ cp -a /home /dev/sdc1 3.) The files I want to copy are in the size of some GB each. So the cache isnt big enough to hold ALL files for the second part. Run the two copies simultaneously, start the first, switch to another tab, start the second. That way the data for the second copy is always the most recently cached. However, I expect the speed limit here may be the USB bus unless you are using USB 3.0 drives on different buses. -- Neil Bothwick I'll never forget the 1st time I ran Windows, but I'm trying... signature.asc Description: PGP signature
Re: [gentoo-user] Copying data efficiently
On Sat, 17 May 2014, at 8:46 am, Mick michaelkintz...@gmail.com wrote: ... a) Unlike other commands, pv will give you a progress bar so that you know how long your back up is taking. I've been using ddrescue recently - by default it seems to be faster than plain old regular dd. I assume it must be possible to adjust the parameters (blocksize?) to speed dd up, but I just want to run the command and not have to bother tinkering. On dumb blind copies, I'm finding ddrescue about 3x or 4x faster than dd, I would estimate. Stroller.
Re: [gentoo-user] Copying data efficiently
Am 17.05.2014 11:58, schrieb Neil Bothwick: On Sat, 17 May 2014 09:59:08 +0200, meino.cra...@gmx.de wrote: 3. Or you could use a sequential copy: cp -a /home /dev/sdb1/ cp -a /home /dev/sdc1 3.) The files I want to copy are in the size of some GB each. So the cache isnt big enough to hold ALL files for the second part. Run the two copies simultaneously, start the first, switch to another tab, start the second. That way the data for the second copy is always the most recently cached. However, I expect the speed limit here may be the USB bus unless you are using USB 3.0 drives on different buses. I was thinking about how to make sure cache is used, and that two simultanious cp won't work, because the progress for the two cp will quickly diverge. But then I realized: there is no need to think about the read cache - the limiting factor is always the writing side, especially with USB! So IMO it doesn't matter at all how you do it! I guess two simultaneous cp will be the same as two sequential cp, except if you have two separate USB-buses. Usually you have just one externally connectible, use lsusb -t to check. If you have less that 2 times the size of your files, IMO simultaneous cp will be worse, because Linux (don't know if USB-subsystem or cp) creates big buffers when cp'ing (check with free -m), and you'll probably get into memory trouble. Greetings, Daniel PS: Quickest way is always to open USB-case and plug SATA cable from motherboard into drive. With 80GB it's always worth the trouble. -- Get my PGP key at: * http://keyserver.ubuntu.com:11371/pks/lookup?op=getsearch=0x837FB8B5BB9D4887 * $ gpg --recv-keys --keyserver keyserver.ubuntu.com 0xBB9D4887 signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Copying data efficiently
Daniel Troeder dan...@admin-box.com [14-05-17 16:36]: Am 17.05.2014 11:58, schrieb Neil Bothwick: On Sat, 17 May 2014 09:59:08 +0200, meino.cra...@gmx.de wrote: 3. Or you could use a sequential copy: cp -a /home /dev/sdb1/ cp -a /home /dev/sdc1 3.) The files I want to copy are in the size of some GB each. So the cache isnt big enough to hold ALL files for the second part. Run the two copies simultaneously, start the first, switch to another tab, start the second. That way the data for the second copy is always the most recently cached. However, I expect the speed limit here may be the USB bus unless you are using USB 3.0 drives on different buses. I was thinking about how to make sure cache is used, and that two simultanious cp won't work, because the progress for the two cp will quickly diverge. But then I realized: there is no need to think about the read cache - the limiting factor is always the writing side, especially with USB! So IMO it doesn't matter at all how you do it! I guess two simultaneous cp will be the same as two sequential cp, except if you have two separate USB-buses. Usually you have just one externally connectible, use lsusb -t to check. If you have less that 2 times the size of your files, IMO simultaneous cp will be worse, because Linux (don't know if USB-subsystem or cp) creates big buffers when cp'ing (check with free -m), and you'll probably get into memory trouble. Greetings, Daniel PS: Quickest way is always to open USB-case and plug SATA cable from motherboard into drive. With 80GB it's always worth the trouble. -- Get my PGP key at: * http://keyserver.ubuntu.com:11371/pks/lookup?op=getsearch=0x837FB8B5BB9D4887 * $ gpg --recv-keys --keyserver keyserver.ubuntu.com 0xBB9D4887 Hi, thank you /.*/ for /.*/ replies ! :) ...one aspect is missing: The load (that is the I/O on the source hd). If this hd is busy spitting out the data twice, it cannot serve outhe jobs twice as long... H Best regards, mcc PS: On the source system # lsusb -t /: Bus 09.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/4p, 12M /: Bus 08.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/2p, 12M /: Bus 07.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/5p, 12M |__ Port 2: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 2: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M /: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/5p, 12M |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 3: Dev 3, If 0, Class=Audio, Driver=snd-usb-audio, 12M |__ Port 3: Dev 3, If 1, Class=Audio, Driver=snd-usb-audio, 12M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/4p, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/5p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/5p, 480M |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/6p, 480M
Re: [gentoo-user] Copying data efficiently
On Sat, 17 May 2014 14:44:04 +0100, Stroller wrote: a) Unlike other commands, pv will give you a progress bar so that you know how long your back up is taking. I've been using ddrescue recently - by default it seems to be faster than plain old regular dd. I assume it must be possible to adjust the parameters (blocksize?) to speed dd up, but I just want to run the command and not have to bother tinkering. Use the bs option. The default is 512, which is incredibly slow, bs=4k makes a substantial difference. However, I usually use dcfldd these days, whch works out the best block size to use and adds progress reporting. -- Neil Bothwick Never underestimate the bandwidth of a station wagon full of tapes! signature.asc Description: PGP signature
Re: [gentoo-user] Copying data efficiently
On Saturday 17 May 2014 08:59:08 meino.cra...@gmx.de wrote: Mick michaelkintz...@gmail.com [14-05-17 09:48]: On Saturday 17 May 2014 04:33:57 meino.cra...@gmx.de wrote: Hi, is there any tool in the Gentoo portage which may speed up (make it mopre efficient) the following task: On my HD there are data I want to copy to two identical external HDs. These HDs are of the same type/model and each is separately connectable via USB to my PC (...these two of the typical mobile external USB-HDs). Instead of copying the data twice from my PC to eah of the HDs I want to do it once...like I would be able to give the cp-command two instead of one target where to copy two. The result should be two identically populated external HDs, each of them useable/readable without the need to the other one. What tool of the portage tree I able to accomplish this task? Best regards, mcc 1. You could set up the two ext drives as a mirrored RAID and use your copying/tar-ing/dd tool of choice. 2. Or you could use pipe and tee to split the feed into any devices you want, e.g.: pv /dev/sda1 | tee (dd of=/dev/sdb1) (dd of=/dev/sdc1) 3. Or you could use a sequential copy: cp -a /home /dev/sdb1/ cp -a /home /dev/sdc1 NOTES: a) Unlike other commands, pv will give you a progress bar so that you know how long your back up is taking. b) The 3rd example above will copy sequentially, but depending on the size of the file(s) the second copy may be read from cache. c) If you're doing this over the network then you can use nc in listening mode at the receiving end, or ssh if the network is untrusted. I'm interested to see what other ways will be suggested for this task. Hi Mick, thank your reply! :) From your numbering of the possibilities... 1.) ...I am no RAID guru and would try this later with data, which are not valuable... If the drives are not permanently connected, then you will have to assemble the RAID each time you want to copy something, after you plug them in. Perhaps not the most efficient method. 2.) That looks interesting! Unfortunately it seems to copy device contents on low level instead of files. The source are directory structure -- not whole devices... Well, I just offered an example for a whole drive. You will need to point it to the file(s) you want to copy over. The pv command works like cat; e.g. cat myfile | tee (blah ...) is the same like: pv myfile | tee (blah ...) Or did I overlook an option mentioned in the manpage...? 3.) The files I want to copy are in the size of some GB each. So the cache isnt big enough to hold ALL files for the second part. Best regards, mcc -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Copying data efficiently
On Sat, 17 May 2014 20:41:15 +0100, Mick wrote: From your numbering of the possibilities... 1.) ...I am no RAID guru and would try this later with data, which are not valuable... If the drives are not permanently connected, then you will have to assemble the RAID each time you want to copy something, after you plug them in. Perhaps not the most efficient method. Whatever method is used will likely consist of multiple commands, and probably a good few arguments, lending itself best to scripting. In that case, the RAID approach becomes no more difficult to use once set up. -- Neil Bothwick The Computer is the logical advancement of humankind: intelligence without morality. signature.asc Description: PGP signature
[gentoo-user] Copying data efficiently
Hi, is there any tool in the Gentoo portage which may speed up (make it mopre efficient) the following task: On my HD there are data I want to copy to two identical external HDs. These HDs are of the same type/model and each is separately connectable via USB to my PC (...these two of the typical mobile external USB-HDs). Instead of copying the data twice from my PC to eah of the HDs I want to do it once...like I would be able to give the cp-command two instead of one target where to copy two. The result should be two identically populated external HDs, each of them useable/readable without the need to the other one. What tool of the portage tree I able to accomplish this task? Best regards, mcc
