Re: copying /dev/da0 with dd(1) to file: output differs
On Sat, Nov 20, 2010 at 01:21:57AM +0200, Peter Pentchev wrote: On Fri, Nov 19, 2010 at 06:35:19PM +0100, Matthias Apitz wrote: El día Friday, November 19, 2010 a las 06:21:09PM +0100, Hans Petter Selasky escribió: Can you dump the data into hex using hexdump -C and show us the difference. Note: the output of the dd(1) is around 3.8 GByte. I compared the 1st 2.000.000 lines of the hexdump: no diff; any better tool to show the 1st block which differs? Usually you would use bs=65536 (Does that change anything)? Same result: they differ :-( matthias bsdiff ? This will not work with such big files (requires 8x memory of the file): I was thinking in a tool just reading each file block by block, comparing the blocks and noting the 1st diff with block offset number. (some 10 lines of C code :-)) E... are you thinking of cmp(1), maybe with the -l option? :) And just for the record, at http://devel.ringlet.net/textproc/block-summarize/ or git://gitorious.org/block-summarize/block-summarize.git you may find a simple utility that summarizes cmp(1)'s output per block. (oof, I just found out that I've still left the default blocksize at 32; will change that for version 0.02, for now there's the -b option :) G'luck, Peter -- Peter Pentchev r...@space.bgr...@ringlet.netr...@freebsd.org PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 Nostalgia ain't what it used to be. signature.asc Description: Digital signature
Re: copying /dev/da0 with dd(1) to file: output differs
El día Friday, November 19, 2010 a las 08:51:38PM +0200, Alexander Motin escribió: Could it be that unwritten/unformatted blocks are read as random data from that USB key? Should I overwrite the full USB key from /dev/zero? It is allowed behavior for SATA SSDs with TRIM command support. Deleted blocks can be not guarantied to return any predictable or even repeatable value. May be this logic could be extended to USB devices. A cmp(1) with -l gives a file like this: $ head -10 diff 196971771 134 34 196971797 134 34 196971833 134 34 196971845 134 34 196971849 134 34 196971875 134 34 196971901 134 34 196971945 134 34 196971955 134 34 196972210 134 34 of 2385059 lines, i.e. of 2385059 bytes which differ in the 3.8 GByte copy; there are only a few patterns of diffs which are repeating: $ sed 's/ */ /g' diff | cut -d' ' -f2,3 | sort -u 0 10 0 104 0 110 0 200 0 204 0 214 0 220 0 234 0 260 0 4 10 0 100 0 110 0 134 0 134 14 134 34 134 4 14 0 200 260 204 234 204 4 214 234 220 260 304 204 34 0 4 0 4 104 4 234 44 0 44 4 (the numbers are in octal). What does this mean? Does not look like errors, in case of random error it whould be more caotic, or? matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
On Sat, 20 Nov 2010, Matthias Apitz wrote: of 2385059 lines, i.e. of 2385059 bytes which differ in the 3.8 GByte copy; there are only a few patterns of diffs which are repeating: $ sed 's/ */ /g' diff | cut -d' ' -f2,3 | sort -u 0 10 0 104 0 110 0 200 0 204 0 214 0 220 0 234 0 260 0 4 10 0 100 0 110 0 134 0 134 14 134 34 134 4 14 0 200 260 204 234 204 4 214 234 220 260 304 204 34 0 4 0 4 104 4 234 44 0 44 4 (the numbers are in octal). What does this mean? Does not look like errors, in case of random error it whould be more caotic, or? Could you try another USB key to determine whether the error is with the key or the host? ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
Matthias Apitz g...@unixarea.de wrote: El d?a Friday, November 19, 2010 a las 08:51:38PM +0200, Alexander Motin escribi?: Could it be that unwritten/unformatted blocks are read as random data from that USB key? Should I overwrite the full USB key from /dev/zero? It is allowed behavior for SATA SSDs with TRIM command support. Deleted blocks can be not guarantied to return any predictable or even repeatable value. May be this logic could be extended to USB devices. A cmp(1) with -l gives a file like this: $ head -10 diff 196971771 134 34 196971797 134 34 ... of 2385059 lines, i.e. of 2385059 bytes which differ in the 3.8 GByte copy; there are only a few patterns of diffs which are repeating: ... What does this mean? Does not look like errors, in case of random error it whould be more caotic, or? Good question. To help sort out whether it matters, what happens if you read the stick using dump(8) instead of dd (or mount it read-only and read it with tar)? Either of those methods will read only the parts that are in use. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
El día Saturday, November 20, 2010 a las 04:19:48PM -0800, per...@pluto.rain.com escribió: ... What does this mean? Does not look like errors, in case of random error it whould be more caotic, or? Good question. To help sort out whether it matters, what happens if you read the stick using dump(8) instead of dd (or mount it read-only and read it with tar)? Either of those methods will read only the parts that are in use. Good idea! A dump is fine and gives more or less the same size as the dump which was restored to the USB key: # dump -0au -f reRead.dmp /dev/da0s1a DUMP: Date of this level 0 dump: Sun Nov 21 08:05:25 2010 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping /dev/da0s1a to reRead.dmp DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 3282758 tape blocks. DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] DUMP: 56.62% done, finished in 0:03 at Sun Nov 21 08:14:18 2010 DUMP: DUMP: 3287561 tape blocks on 1 volume DUMP: finished in 586 seconds, throughput 5610 KBytes/sec DUMP: level 0 dump on Sun Nov 21 08:05:25 2010 DUMP: Closing reRead.dmp DUMP: DUMP IS DONE current# ls -l reRead.dmp ~guru/usb9root.dmp -r--r--r-- 1 guru wheel 3351920640 12 nov 13:40 /home/guru/usb9root.dmp -rw-r--r-- 1 root wheel 3366461440 21 nov 08:15 reRead.dmp One could as well MD5 compare all files with find(1), but this is perhaps a bit overkill... I will buy a 2nd key and restore the dd(1)'ed file to it with dd(1)... matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
copying /dev/da0 with dd(1) to file: output differs
Hello, How is this possible: $ dd if=/dev/da0 of=f1 7892087+0 records in 7892087+0 records out $ dd if=/dev/da0 of=f2 7892087+0 records in 7892087+0 records out $ diff f1 f1 Files f1 and f2 differ (and no, /dev/da0s1a is not mounted). Thx matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
On Friday 19 November 2010 15:33:37 Matthias Apitz wrote: Hello, How is this possible: $ dd if=/dev/da0 of=f1 7892087+0 records in 7892087+0 records out $ dd if=/dev/da0 of=f2 7892087+0 records in 7892087+0 records out $ diff f1 f1 Files f1 and f2 differ (and no, /dev/da0s1a is not mounted). Thx matthias Hi, Can you dump the data into hex using hexdump -C and show us the difference. Usually you would use bs=65536 (Does that change anything)? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
El día Friday, November 19, 2010 a las 03:37:25PM +0100, Hans Petter Selasky escribió: On Friday 19 November 2010 15:33:37 Matthias Apitz wrote: Hello, How is this possible: $ dd if=/dev/da0 of=f1 7892087+0 records in 7892087+0 records out $ dd if=/dev/da0 of=f2 7892087+0 records in 7892087+0 records out $ diff f1 f1 Files f1 and f2 differ (and no, /dev/da0s1a is not mounted). Thx matthias Hi, Can you dump the data into hex using hexdump -C and show us the difference. Note: the output of the dd(1) is around 3.8 GByte. I compared the 1st 2.000.000 lines of the hexdump: no diff; any better tool to show the 1st block which differs? Usually you would use bs=65536 (Does that change anything)? Same result: they differ :-( matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
El día Friday, November 19, 2010 a las 06:21:09PM +0100, Hans Petter Selasky escribió: Can you dump the data into hex using hexdump -C and show us the difference. Note: the output of the dd(1) is around 3.8 GByte. I compared the 1st 2.000.000 lines of the hexdump: no diff; any better tool to show the 1st block which differs? Usually you would use bs=65536 (Does that change anything)? Same result: they differ :-( matthias bsdiff ? This will not work with such big files (requires 8x memory of the file): I was thinking in a tool just reading each file block by block, comparing the blocks and noting the 1st diff with block offset number. (some 10 lines of C code :-)) matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
On Friday 19 November 2010 18:35:19 Matthias Apitz wrote: El día Friday, November 19, 2010 a las 06:21:09PM +0100, Hans Petter Selasky escribió: Can you dump the data into hex using hexdump -C and show us the difference. Note: the output of the dd(1) is around 3.8 GByte. I compared the 1st 2.000.000 lines of the hexdump: no diff; any better tool to show the 1st block which differs? Usually you would use bs=65536 (Does that change anything)? Same result: they differ :-( matthias bsdiff ? This will not work with such big files (requires 8x memory of the file): I was thinking in a tool just reading each file block by block, comparing the blocks and noting the 1st diff with block offset number. (some 10 lines of C code :-)) matthias Maybe you need to write a small C-program to do that. You can use bcmp() to compare two buffers. --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
El día Friday, November 19, 2010 a las 07:16:53PM +0100, Hans Petter Selasky escribió: I was thinking in a tool just reading each file block by block, comparing the blocks and noting the 1st diff with block offset number. (some 10 lines of C code :-)) matthias Maybe you need to write a small C-program to do that. You can use bcmp() to compare two buffers. Will do that tomorrow. Just an idea: The USB key in question was new and I only created the file system on it the usual way (fdisk, bsdlabel, newfs). Then I restored the dump on it (which took 26 hours for 3.1 GByte dump file). The USB key boots fine, btw. Could it be that unwritten/unformatted blocks are read as random data from that USB key? Should I overwrite the full USB key from /dev/zero? matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
On 19.11.2010 20:39, Matthias Apitz wrote: El día Friday, November 19, 2010 a las 07:16:53PM +0100, Hans Petter Selasky escribió: I was thinking in a tool just reading each file block by block, comparing the blocks and noting the 1st diff with block offset number. (some 10 lines of C code :-)) matthias Maybe you need to write a small C-program to do that. You can use bcmp() to compare two buffers. Will do that tomorrow. Just an idea: The USB key in question was new and I only created the file system on it the usual way (fdisk, bsdlabel, newfs). Then I restored the dump on it (which took 26 hours for 3.1 GByte dump file). The USB key boots fine, btw. Could it be that unwritten/unformatted blocks are read as random data from that USB key? Should I overwrite the full USB key from /dev/zero? It is allowed behavior for SATA SSDs with TRIM command support. Deleted blocks can be not guarantied to return any predictable or even repeatable value. May be this logic could be extended to USB devices. -- Alexander Motin ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
On Fri, 19 Nov 2010, Matthias Apitz wrote: El d?a Friday, November 19, 2010 a las 07:16:53PM +0100, Hans Petter Selasky escribi?: I was thinking in a tool just reading each file block by block, comparing the blocks and noting the 1st diff with block offset number. (some 10 lines of C code :-)) matthias Maybe you need to write a small C-program to do that. You can use bcmp() to compare two buffers. Will do that tomorrow. Just an idea: The USB key in question was new and I only created the file system on it the usual way (fdisk, bsdlabel, newfs). Then I restored the dump on it (which took 26 hours for 3.1 GByte dump file). The USB key boots fine, btw. 26 *hours*? USB 1.1? Could it be that unwritten/unformatted blocks are read as random data from that USB key? Well, they could be anything. But the original filesystem has formerly-used blocks with old data from deleted files. dump doesn't copy those, but dd does. Should I overwrite the full USB key from /dev/zero? Possibly there would still be differences. Filesystem metadata like date last mounted, for example. If you want a block-by-block duplicate, the brute-force method is to just dd the whole drive. Use bs=64k or bs=1m to help reduce overhead. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
El día Friday, November 19, 2010 a las 12:39:13PM -0700, Warren Block escribió: Just an idea: The USB key in question was new and I only created the file system on it the usual way (fdisk, bsdlabel, newfs). Then I restored the dump on it (which took 26 hours for 3.1 GByte dump file). The USB key boots fine, btw. 26 *hours*? USB 1.1? I don't think so. It is the Acer One D250 netbook. Well, they could be anything. But the original filesystem has formerly-used blocks with old data from deleted files. dump doesn't copy those, but dd does. Should I overwrite the full USB key from /dev/zero? Possibly there would still be differences. Filesystem metadata like date last mounted, for example. If you want a block-by-block duplicate, the brute-force method is to just dd the whole drive. Use bs=64k or bs=1m to help reduce overhead. Warren, perhaps you missed my point. I have a prepared boot-able key and I want to give away a copy of it as a file on DVD. So I dd(1)'ed the key to disk and did this twice to ensure that the copy was fine, but the two files differ. How can I make sure that the file on disk (or DVD) is a exact copy of the key? matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
On Fri, Nov 19, 2010 at 06:35:19PM +0100, Matthias Apitz wrote: El día Friday, November 19, 2010 a las 06:21:09PM +0100, Hans Petter Selasky escribió: Can you dump the data into hex using hexdump -C and show us the difference. Note: the output of the dd(1) is around 3.8 GByte. I compared the 1st 2.000.000 lines of the hexdump: no diff; any better tool to show the 1st block which differs? Usually you would use bs=65536 (Does that change anything)? Same result: they differ :-( matthias bsdiff ? This will not work with such big files (requires 8x memory of the file): I was thinking in a tool just reading each file block by block, comparing the blocks and noting the 1st diff with block offset number. (some 10 lines of C code :-)) E... are you thinking of cmp(1), maybe with the -l option? :) G'luck, Peter -- Peter Pentchev r...@space.bgr...@ringlet.netr...@freebsd.org PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 When you are not looking at it, this sentence is in Spanish. signature.asc Description: Digital signature
Re: copying /dev/da0 with dd(1) to file: output differs
On Fri, 19 Nov 2010, Matthias Apitz wrote: El d?a Friday, November 19, 2010 a las 12:39:13PM -0700, Warren Block escribi?: Should I overwrite the full USB key from /dev/zero? Possibly there would still be differences. Filesystem metadata like date last mounted, for example. If you want a block-by-block duplicate, the brute-force method is to just dd the whole drive. Use bs=64k or bs=1m to help reduce overhead. Warren, perhaps you missed my point. I have a prepared boot-able key and I want to give away a copy of it as a file on DVD. So I dd(1)'ed the key to disk and did this twice to ensure that the copy was fine, but the two files differ. How can I make sure that the file on disk (or DVD) is a exact copy of the key? Okay, that makes more sense. Did you mount the key in between the two times it was dd-ed? ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: copying /dev/da0 with dd(1) to file: output differs
El día Friday, November 19, 2010 a las 05:42:57PM -0700, Warren Block escribió: On Fri, 19 Nov 2010, Matthias Apitz wrote: El d?a Friday, November 19, 2010 a las 12:39:13PM -0700, Warren Block escribi?: Should I overwrite the full USB key from /dev/zero? Possibly there would still be differences. Filesystem metadata like date last mounted, for example. If you want a block-by-block duplicate, the brute-force method is to just dd the whole drive. Use bs=64k or bs=1m to help reduce overhead. Warren, perhaps you missed my point. I have a prepared boot-able key and I want to give away a copy of it as a file on DVD. So I dd(1)'ed the key to disk and did this twice to ensure that the copy was fine, but the two files differ. How can I make sure that the file on disk (or DVD) is a exact copy of the key? Okay, that makes more sense. Did you mount the key in between the two times it was dd-ed? No. I just fired up the dd(1) a 2nd time to another file to check if it would produce two times the same result. And, surprise, they differ, they differ each time and in laptops differentes as well. matthias -- Matthias Apitz t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e g...@unixarea.de - w http://www.unixarea.de/ ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
