Re: [Dorset] Using shred
Hi Terry, > > /dev/urandom can be quite slow for large amounts. > > OK. I'll try the second drive with zeros. No, I'd stick to /dev/urandom if you don't want to do that cat(1) method since it's slow, but probably not as slow as writing over USB 2.0 to spinning rust. Who knows, perhaps the drive optimises writes of zeroes to a large run of sectors by setting a bit somewhere for that run. :-) I was pointing it out for when writing to something quicker and warning /dev/urandom *can* be a bottleneck. When the first one of these is running, dstat(1) shows one core is 100% busy in `sys', doing the PRNG. Compare the usecs/call. $ strace -ce read,write dd if=/dev/urandom of=/dev/null bs=1M count=1K 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.8773 s, 83.4 MB/s % time seconds usecs/call callserrors syscall -- --- --- - - 99.79 12.466984 12139 1027 read 0.210.025800 25 1027 write -- --- --- - - 100.00 12.492784 2054 total $ $ strace -ce read,write dd if=/dev/zero of=/dev/null bs=1M count=1K 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.81163 s, 1.3 GB/s % time seconds usecs/call callserrors syscall -- --- --- - - 95.580.545116 531 1027 read 4.420.025235 25 1027 write -- --- --- - - 100.000.570351 2054 total $ > > $ units -1v 22GiB/40minutes hour/TB > > reciprocal conversion > > 1 / (22GiB/40minutes) = 28.221896 hour/TB > > $ > > Yeah, see my other post relating to my arithmetic ;-( Yes, I was showing how to avoid that arithmetic. :-) Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Sunday, 18 February 2018 11:16:04 GMT Ralph Corderoy wrote: > dd(1) is quick. It read(2)s a block of bytes and write(2)s that block. > The kernel does the transfer of bytes from the device to dd's memory and > vica versa. If you don't choose a block size then it might be quite > small, > > > and most of your time is spent in overhead of switching between dd and > the kernel. Using bs=1M would cut down that overhead as you're unlikely > to be using a device that insists on a particular block size. I'm now halfway through writing the first drive, so I'll let that finish at around 3 pm today and try the second drive with a BS of 1M. > /dev/urandom can be quite slow for large amounts. OK. I'll try the second drive with zeros. > One can go faster still by cutting out a read(2) for every write(2) by > having a little Perl script or C program that loops, flinging the same > data read once into every write(). Thanks. I think I'll keep it simple though :-) > > It seems to be pretty quick having reached 22 GB done in around 40 > > minutes. > > I'm assuing that's GiB to the drive's TB. > > $ units -1v 22GiB/40minutes hour/TB > reciprocal conversion > 1 / (22GiB/40minutes) = 28.221896 hour/TB > $ Yeah, see my other post relating to my arithmetic ;-( -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
Hi Terry, > I think that it's more likely to be the interface. The drive is a > Seagate ST31000523AS, which has a SATA speed of 6 GB/s, a Transfer > Speed of 600 MB/s and a Sustained Data Rate of 125 MB/s. I'd expect read to be different to write? And if they only give one figure then it's the better looking, read, one. https://www.seagate.com/staticfiles/support/disc/manuals/desktop/Barracuda%207200.12/100529369a.pdf says average seek time differs for read and write, presumably the former doesn't need to be positioned so accurately so less settle time is required. > However, there is a USB interface in the way and I think that is > slowing the process down by a factor of 4 to 5. Especially if there's USB 2.0 involved rather than 3.0. Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
Hi Terry, > The above link also suggests that dd is the longest method. dd(1) is quick. It read(2)s a block of bytes and write(2)s that block. The kernel does the transfer of bytes from the device to dd's memory and vica versa. If you don't choose a block size then it might be quite small, $ dd if=/dev/zero of=/dev/null count=1 1+0 records in 1+0 records out 512 bytes copied, 0.000276964 s, 1.8 MB/s $ and most of your time is spent in overhead of switching between dd and the kernel. Using bs=1M would cut down that overhead as you're unlikely to be using a device that insists on a particular block size. > I must say that I used to get exceeding bored when copying a 4 GB SD > Card. The destination media is the bottleneck there. > However, I tried it and then realised that I would have to wait for it > to finish before I found out the duration dd(1) says to send a USR1 signal. The arrows are the lines I typed. The bulleted lines are the response to the signal. $ dd bs=2 → foo foo • 2+0 records in • 2+0 records out • 4 bytes copied, 4.40433 s, 0.0 kB/s → bar bar 4+0 records in 4+0 records out 8 bytes copied, 7.53309 s, 0.0 kB/s $ > sudo dcfldd if=/dev/urandom of=/dev/sdc /dev/urandom can be quite slow for large amounts. $ dd if=/dev/urandom bs=1M count=1K of=/dev/null 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.6809 s, 84.7 MB/s $ $ openssl rand $((2**20)) >rnd $ r=rnd $ r="$r $r $r $r" $ r="$r $r $r $r" $ r="$r $r $r $r" $ r="$r $r $r $r" $ r="$r $r $r $r" $ cat $r | wc -c 1073741824 $ echo $((2**30)) 1073741824 $ while cat $r; do :; done | > dd iflag=fullblock bs=1M count=1K of=/dev/null 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.04574 s, 353 MB/s $ One can go faster still by cutting out a read(2) for every write(2) by having a little Perl script or C program that loops, flinging the same data read once into every write(). > It seems to be pretty quick having reached 22 GB done in around 40 > minutes. I'm assuing that's GiB to the drive's TB. $ units -1v 22GiB/40minutes hour/TB reciprocal conversion 1 / (22GiB/40minutes) = 28.221896 hour/TB $ Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
Try Again. This seems to happen when people cc me as well as sending the response to the list. My mail client helpfully suppresses the original and only sends me the duplicate, which I then reply to On Sunday, 18 February 2018 09:49:55 GMT PeterMerchant via dorset wrote: > I have an idle PC sitting in my garage. You are welcome to borrow it to > install these disks and wipe them, if you think that would be faster. I > think it has both SATA and an IDE drive, but I would need to check. > That would get over the USB bottleneck. Peter, Thanks for the offer, but I'm now 25% of the way through the first drive. Now that I've decided that one pass of random data will suffice, I'm quite happy to do one drive today and one tomorrow. It was the overnight thing that was the problem. If the PC had been in any room but this one, I'd have let it run overnight, but I don't have enough space anywhere else. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Sunday, 18 February 2018 08:48:00 GMT Hamish MB wrote: > You could well be being limited by your disks speed then. 100GB per hours is > pretty fast for a HDD. Can you connect two at once to speed it up? I think that it's more likely to be the interface. The drive is a Seagate ST31000523AS, which has a SATA speed of 6 GB/s, a Transfer Speed of 600 MB/s and a Sustained Data Rate of 125 MB/s. Clearly the Data Rate would be the limiting factor if I was connecting the drive directly to a SATA interface so it would theoretically give a total time of 2.22 hours to complete the process. However, there is a USB interface in the way and I think that is slowing the process down by a factor of 4 to 5. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On 18/02/18 08:48, Hamish MB wrote: You could well be being limited by your disks speed then. 100GB per hours is pretty fast for a HDD. Can you connect two at once to speed it up? Hamish On 18 Feb 2018, at 08:45, Terry Coles mailto:d-...@hadrian-way.co.uk>> wrote: On Sunday, 18 February 2018 07:59:30 GMT Terry Coles wrote: It seems to be pretty quick having reached 22 GB done in around 40 minutes. Of course, if I was any good at basic arithmetic, I would have known that this is no quicker than shred. It just passed 100 GB after about an hour; 1000 GB = 10 hours. (I must have got the time wrong when I estimated earlier (as well!)) Even so, I think one pass of this command or shred should suffice. I have an idle PC sitting in my garage. You are welcome to borrow it to install these disks and wipe them, if you think that would be faster. I think it has both SATA and an IDE drive, but I would need to check. That would get over the USB bottleneck. Peter -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
You could well be being limited by your disks speed then. 100GB per hours is pretty fast for a HDD. Can you connect two at once to speed it up? Hamish On 18 Feb 2018, at 08:45, Terry Coles mailto:d-...@hadrian-way.co.uk>> wrote: On Sunday, 18 February 2018 07:59:30 GMT Terry Coles wrote: It seems to be pretty quick having reached 22 GB done in around 40 minutes. Of course, if I was any good at basic arithmetic, I would have known that this is no quicker than shred. It just passed 100 GB after about an hour; 1000 GB = 10 hours. (I must have got the time wrong when I estimated earlier (as well!)) Even so, I think one pass of this command or shred should suffice. -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Sunday, 18 February 2018 07:59:30 GMT Terry Coles wrote: > It seems to be pretty quick having reached 22 GB done in around 40 minutes. Of course, if I was any good at basic arithmetic, I would have known that this is no quicker than shred. It just passed 100 GB after about an hour; 1000 GB = 10 hours. (I must have got the time wrong when I estimated earlier (as well!)) Even so, I think one pass of this command or shred should suffice. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Saturday, 17 February 2018 23:56:31 GMT Ralph Corderoy wrote: > The NAS offered a `reformat'? How long did it take? It's part of the build process. The NAS offers Raid 0, Raid 1, etc and as part of the process formats the drives. It seems to be a special format in D-Link boxes, because Kubuntu didn't recognise the format. > Kill it off then start from scratch tomorrow morning, assuming ten hours > for a disk? Actually 30 hours for disc, 10 hours for a pass. The command that I used does three passes. > Drives have an internal `erase' method, but over USB probably isn't a > good way to play with it. > https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase There are enough warnings there to sink a battleship. I'll stick to traditional methods. > Re-assemble the NAS, see the empty (mirrored?) drives, copy innocuous > stuff onto it over and over until `full'. `Erase' again for sale? It also takes a very long time to copy stuff onto the drive from a PC on the network. (This is one reason why I upgraded to a Netgear ReadyNAS. The main reason was that the 1 TB drives were full, so I've bought a box with two 3 TB drives included and much better performance.) I suppose that I could copy a file containing something boring like 'foo bah' repeatedly through a script running locally on the NAS box, except that this box doesn't allow remote login other than through the browser or SMB. I'm currently running sudo dcfldd if=/dev/urandom of=/dev/sdc, see my post to Hamish. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Saturday, 17 February 2018 19:22:05 GMT Hamish MB wrote: > For modern drives a single pass with zeros is usually just fine. You can use > dd if=/dev/zero of=/Dev/sdxy > You could copy from /Dev/urandom for a quicker random number pass too. Hamish, I did find some information that told me about alternatives at http://daniemon.com/blog/how-to-wipe-a-hard-drive-with-linux/ The quickest method seemed to be 'wipe' (by a whisker according to the link), but the ETA given when I tried the command was 6 days and 5 hours! The above link also suggests that dd is the longest method. I must say that I used to get exceeding bored when copying a 4 GB SD Card. However, I tried it and then realised that I would have to wait for it to finish before I found out the duration, so I've installed dcfldd which is currently running using the command sudo dcfldd if=/dev/urandom of=/dev/sdc. It seems to be pretty quick having reached 22 GB done in around 40 minutes. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
Hi Terry, > I have reformatted the drives in-situ The NAS offered a `reformat'? How long did it take? > My PC is in the bedroom, so I'd rather not run it overnight. Kill it off then start from scratch tomorrow morning, assuming ten hours for a disk? > I suspect that part of the problem is that I'm putting the drive into > an external USB adaptor, but my PC has no spare slots for other > drives. Yep. Drives have an internal `erase' method, but over USB probably isn't a good way to play with it. https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase Re-assemble the NAS, see the empty (mirrored?) drives, copy innocuous stuff onto it over and over until `full'. `Erase' again for sale? Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
For modern drives a single pass with zeros is usually just fine. You can use dd if=/dev/zero of=/Dev/sdxy You could copy from /Dev/urandom for a quicker random number pass too. Hamish On 17 Feb 2018, at 18:38, Keith Edmunds mailto:k...@midnighthax.com>> wrote: On Sat, 17 Feb 2018 18:31:53 +, d-...@hadrian-way.co.uk said: I'd be happier if there was no chance of data recovery. Then destroy the drives. -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Saturday, 17 February 2018 18:36:34 GMT Keith Edmunds wrote: > On Sat, 17 Feb 2018 18:31:53 +, d-...@hadrian-way.co.uk said: > > I'd be happier if there was no chance of data recovery. > > Then destroy the drives. I want to sell them with the NAS Box. I do know about hammers :-) -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Sat, 17 Feb 2018 18:31:53 +, d-...@hadrian-way.co.uk said: > I'd be happier if there was no chance of data recovery. Then destroy the drives. -- "Laughter is the best medicine, though it tends not to work in the case of impotence" - Jo Brand -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
On Saturday, 17 February 2018 17:56:45 GMT Keith Edmunds wrote: > Shred is slow. No one will trivially read the ex-data following a > reformat. I suppose if the drive used to hold your plans for world > domination, a few more hours of shred is a small price to pay. However, > if the drive only held data of interest to you and you don't want the PC > running overnight, don't worry about it. I'm sure you are right, but the NAS Box was used to back up four different PCs and I'm not sure what was there. My two machines were both Linux and I only backed up the /home partition. Even so, there could be stuff that might give passwords away etc. The other two PCs were both Windows machines (son and daughter's) and they both tended to back up the whole of their 'C'-Drives, so it's anyone's guess what might be buried in the Registry. I'd be happier if there was no chance of data recovery. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Using shred
Shred is slow. No one will trivially read the ex-data following a reformat. I suppose if the drive used to hold your plans for world domination, a few more hours of shred is a small price to pay. However, if the drive only held data of interest to you and you don't want the PC running overnight, don't worry about it. -- "Never, never, never give up" - Winston Churchill -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
[Dorset] Using shred
Hi, I am retiring my old D-Link Sharecenter NAS with a view to flogging it on eBay. Clearly, I want to ensure the data has been completely scrubbed off the two 1TB drives. I have reformatted the drives in-situ, but felt that I should do more than that, so I've removed the drives and am currently running shred on the first, using : sudo shred -f -v /dev/sdc The trouble is that the whole thing is so slw! I'm currently at 10% on the first pass after an hour. At that rate the first pass won't be completed until 3 am tomorrow and the third pass sometime tomorrow evening! My PC is in the bedroom, so I'd rather not run it overnight. I suspect that part of the problem is that I'm putting the drive into an external USB adaptor, but my PC has no spare slots for other drives. Any suggestions? -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2018-03-06 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
