Re: fastest raw device copy?
On Mon, Nov 03, 2008 at 09:04:47AM +0100, Wojciech Puchar wrote: Would there be anything wrong in cat /dev/ad0 /dev/da0 ? small block size, IMHO 4kB with cat Not to mention, doesn't cat operate on a character level and not a block level? -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
Would there be anything wrong in cat /dev/ad0 /dev/da0 ? small block size, IMHO 4kB with cat ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
? small block size, IMHO 4kB with cat Not to mention, doesn't cat operate on a character level and not a block level? it just do 4kB read. that's all ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 On Fri, 31 Oct 2008 14:16:38 +0100 Ivan Voras [EMAIL PROTECTED] wrote: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m Would there be anything wrong in cat /dev/ad0 /dev/da0 ? - -- Nikola Lečić = Никола Лечић fingerprint : FEF3 66AF C90E EDC3 D878 7CDC 956D F4AB A377 1C9B -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (FreeBSD) iJwEAQEDAAYFAkkOQCsACgkQ/MM/0rYIoZjjHQQAhXi6gu5JhJjzZmE97buiZ15u Q6AT+qvAja6cALfGAGVrzJEzljfcbe3PnBdOnn1CTYbS62EHaVWLnvOKRGrpvzFE q/WxQ9qCRfcsSx3o6eKxfTM6d4b92ZP+d1iPotmzutQl8TbxlJxNTP9i2b6cDw6a au6zdoApH5A6UxyaJA8= =oFoQ -END PGP SIGNATURE- ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
fastest raw device copy?
Hi list, I'm considering using a bootable USB stick with FreeBSD to perform a backup of my notebooks' 500 GB hard disk to a physically identical (same make, same type, same size) hard disk attached to USB. What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 but maybe there is a utility which does this faster or a larger buffer size? Probably the limit will be the USB 2.0 bus speed anyway? -- Christoph ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: Hi list, I'm considering using a bootable USB stick with FreeBSD to perform a backup of my notebooks' 500 GB hard disk to a physically identical (same make, same type, same size) hard disk attached to USB. What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 but maybe there is a utility which does this faster or a larger buffer size? Probably the limit will be the USB 2.0 bus speed anyway? In general, what you're doing is correct for a block copy. There is nothing (that I know of) which is faster; you're copying 500GB of data (including the unused portion -- you *did* ask for a block copy), and this takes a long time. Be patient. On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
In general, what you're doing is correct for a block copy. There is nothing (that I know of) which is faster; you're copying 500GB of data (including the unused portion -- you *did* ask for a block copy), and this takes a long time. Be patient. On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. should be big but less than drive's buffer, and be multiply of sector size. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m signature.asc Description: OpenPGP digital signature
Re: fastest raw device copy?
Ivan Voras schrieb: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? -- Christoph ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
Christoph Kukulies schrieb: Ivan Voras schrieb: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? Ack, I mean bytes :) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? systat :vmstat ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
On Fri, Oct 31, 2008 at 03:36:02PM +0100, Christoph Kukulies wrote: Ivan Voras schrieb: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? Just type a ^T on the terminal dd is running on. This will send a SIGINFO to dd which will cause it to print out that information to the terminal. -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
On Fri, Oct 31, 2008 at 03:36:02PM +0100, Christoph Kukulies wrote: Ivan Voras schrieb: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. Six hours? Hmm... That seems too long, but of course the FreeBSD USB stack is involved, and a USB device in general. I would have assumed that copy should have finished after 2-3 hours tops. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? iostat or gstat (I'm willing to bet you prefer the latter) will get you what you want, more or less. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
On Friday 31 October 2008, Christoph Kukulies wrote: Ivan Voras schrieb: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? Press ^T. It will show you progress and I/O speed. -- Pieter de Goeje ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
Christoph Kukulies wrote: OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. No, with a size that isn't a multiple of sector sizes your transferred data will be corrupted. Actually, it's surprising that your number even works - the system should have complained when you requested that size for bs. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? hit Ctrl-T while running dd. signature.asc Description: OpenPGP digital signature
Re: fastest raw device copy?
On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: Hi list, I'm considering using a bootable USB stick with FreeBSD to perform a backup of my notebooks' 500 GB hard disk to a physically identical (same make, same type, same size) hard disk attached to USB. What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 but maybe there is a utility which does this faster or a larger buffer size? Probably the limit will be the USB 2.0 bus speed anyway? Are you sure you want to do a sector-by-sector copy? That won't get you much that is useful in terms of a backup. Can't you use dump/restore instead? Dump each file system on /dev/ad0 to a file on /dev/da0. Create a file system on /dev/da0 using newfs first. You may or may not want to create a FreeBSD slice and partition there before doing the newfs. Make a mount point and mount it. mkdir /bkmnt mount /dev/da0 /bkmntOr if you created slice and partition in /dev/da0 mount /dev/da0s1a /bkmnt Then do the dumps dump 0af /bkmnt/rootbackup / dump 0af /bkmnt/usrbackup /usr dump 0af /bkmnt/homehackup /home etc for whatever file systems you want to back up. You will be much better off than with a sector by sector copy. jerry -- Christoph ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
Jeremy Chadwick schrieb: On Fri, Oct 31, 2008 at 03:36:02PM +0100, Christoph Kukulies wrote: Ivan Voras schrieb: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. Six hours? Hmm... That seems too long, but of course the FreeBSD USB stack is involved, and a USB device in general. I would have assumed that copy should have finished after 2-3 hours tops. An idea how I can check the current amount of transfered byed alongside the running dd command? Or watch the current i/o rate? iostat or gstat (I'm willing to bet you prefer the latter) will get you what you want, more or less. The job just finished and I have a figure of the Ubuntu performance, with the unfortunate blocksize parameter though, so I think it isn't much worth. Anyway here is the figure of the above dd command copying 500GB to a WDC disk in an Icy box. 50010+1 Datensätze ein 50010+1 Datensätze aus 500107862016 Bytes (500GB) kopiert, 25787,9 s, 19,4 MB/s Will do that using FreeBSD next time. Thanks a lot so far. -- Christoph ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
Jerry McAllister schrieb: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: Hi list, I'm considering using a bootable USB stick with FreeBSD to perform a backup of my notebooks' 500 GB hard disk to a physically identical (same make, same type, same size) hard disk attached to USB. What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 but maybe there is a utility which does this faster or a larger buffer size? Probably the limit will be the USB 2.0 bus speed anyway? Are you sure you want to do a sector-by-sector copy? That won't get you much that is useful in terms of a backup. Can't you use dump/restore instead? Dump each file system on /dev/ad0 to a file on /dev/da0. Create a file system on /dev/da0 using newfs first. You may or may not want to create a FreeBSD slice and partition there before doing the newfs. Make a mount point and mount it. mkdir /bkmnt mount /dev/da0 /bkmntOr if you created slice and partition in /dev/da0 mount /dev/da0s1a /bkmnt Then do the dumps dump 0af /bkmnt/rootbackup / dump 0af /bkmnt/usrbackup /usr dump 0af /bkmnt/homehackup /home etc for whatever file systems you want to back up. You will be much better off than with a sector by sector copy. jerry The idea was to have a drop in backup for my notebook that allows me to continue working with a minimum of delay. (requires a philips screwdriver though :-) Of course a failure of the source disk while doing the image copy as the worst case scenario would leave me with empty hands :-) There are a couple of partitions with different OSs on that hard drive. -- Christoph ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
On Fri, Oct 31, 2008 at 04:48:32PM +0100, Christoph Kukulies wrote: Jerry McAllister schrieb: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: Hi list, I'm considering using a bootable USB stick with FreeBSD to perform a backup of my notebooks' 500 GB hard disk to a physically identical (same make, same type, same size) hard disk attached to USB. What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 but maybe there is a utility which does this faster or a larger buffer size? Probably the limit will be the USB 2.0 bus speed anyway? Are you sure you want to do a sector-by-sector copy? That won't get you much that is useful in terms of a backup. Can't you use dump/restore instead? Dump each file system on /dev/ad0 to a file on /dev/da0. Create a file system on /dev/da0 using newfs first. You may or may not want to create a FreeBSD slice and partition there before doing the newfs. Make a mount point and mount it. mkdir /bkmnt mount /dev/da0 /bkmntOr if you created slice and partition in /dev/da0 mount /dev/da0s1a /bkmnt Then do the dumps dump 0af /bkmnt/rootbackup / dump 0af /bkmnt/usrbackup /usr dump 0af /bkmnt/homehackup /home etc for whatever file systems you want to back up. You will be much better off than with a sector by sector copy. jerry The idea was to have a drop in backup for my notebook that allows me to continue working with a minimum of delay. (requires a philips screwdriver though :-) Of course a failure of the source disk while doing the image copy as the worst case scenario would leave me with empty hands :-) There are a couple of partitions with different OSs on that hard drive. Still, that is not really the best way to do backups. For the FreeBSD, you can build the slice and partitions in it and back up each appropriately. Make mount points as needed mkdir /bkroot mkdir /bkusr mkdir /bkhome etc Do the mounts mount /dev/da0s1a /bkroot mount /dev/da0s1d /bkusr mount /dev/da0s1g /bkhome etc as appropriate Make sure those filesystems on the USB are empty. Then use dump/retore to make the copies cd /bkroot dump 0af - / | restore -rf - cd /bkusr dump 0af - /usr | restore -rf - cd /bkhome dump 0af - /home | restore -rf - etc as appropriate This makes a much more reliable backup that the sector by sector thing. If you make the slice bootable and have an MBR, it can be bootable. It takes care of any difference in device size/block counts and dealing with back blocks transparently. If you have more than one slice and each has a different OS, then you need to make an equivalent backup for each OS to the appropriate slice on the USB. jerry -- Christoph ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: fastest raw device copy?
On Fri, Oct 31, 2008 at 08:13:07AM -0700, Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 03:36:02PM +0100, Christoph Kukulies wrote: Ivan Voras schrieb: Jeremy Chadwick wrote: On Fri, Oct 31, 2008 at 09:48:16AM +0100, Christoph Kukulies wrote: What would be the fastest way to do that sector by sector copy? I'm using dd right now, dd if=/dev/ad0 of=/dev/da0 bs=1000 On the flip side, your blocksize (bs) there is quite high for no good reason. I'd pick something more like bs=64k or bs=128k. The default (512) is too small for what you want, but 10MBytes is silly. Not only that, but 1000 isn't even correct - it needs to be a multiple of sector size. Generally, using suffixes will do the right thing: dd if=/dev/ad0 of=/dev/da0 bs=1m OK, I understand that 1000 isn't good, I just thought it wouldn't harm. But if it is a transfer rate killer then I'd better think of typing ^C now. The command is running for 6 hours now. Six hours? Hmm... That seems too long, but of course the FreeBSD USB stack is involved, and a USB device in general. I would have assumed that copy should have finished after 2-3 hours tops. 2-3 hours is about the *minimum* amount of time needed to read all of a large modern disk - and that is when you do the transfer over SATA/IDE. Over USB it will be significantly slower. A modern, fairly fast disk has an average sequential transfer rate of around 60MB/s (higher on the outer tracks, and lower on the innermost track, but the average will come out to about that.) Assume a 500GB disk. Then we get 500GB/(60MB/s) = 500*1000/60 s = 8333s = 2h18min to read all of it. Over USB you probably will not get much more than 20-25MB/s. 20MB/s is exactly a third of the 60MB/s we used earlier gives a total time of 3*(2h18min) ~= 7h. Those 6 hours do not sound unreasonable at all. (All of this assumes just reading the disk from start to end. If you need to seek back and forth it will take even longer.) -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
