Re: unzip parallel, 3x faster than 7zip

2012-04-08 Thread Marco Leise
Am Sat, 07 Apr 2012 21:45:04 +0200 schrieb Jay Norwood j...@prismnet.com: So ... it looks like the defrag helps, as the 109 sec values are at the low end of the range I've seen previously. Still it is totally surprising to me that deleting files should take longer than creating the same

Re: unzip parallel, 3x faster than 7zip

2012-04-08 Thread Jay Norwood
On Sunday, 8 April 2012 at 13:55:21 UTC, Marco Leise wrote: Maybe the kernel caches writes, but synchronizes deletes? (So the seek times become apparent there, and not in the writes) Also check the file creation flags, maybe you can hint Windows to the final file size and they wont be

Re: unzip parallel, 3x faster than 7zip

2012-04-07 Thread Jay Norwood
On Saturday, 7 April 2012 at 05:02:04 UTC, dennis luehring wrote: 7zip took 55 secs _on the same file_. that is ok but he still compares different implementations 7zip is the program. It unzips many formats, with the standard zip format being one of them. The parallel d program is three

Re: unzip parallel, 3x faster than 7zip

2012-04-07 Thread Rainer Schuetze
On 4/7/2012 12:32 AM, Jay Norwood wrote: I got procmon to see what is going on. Win7 has doing indexing and thumbnails, and there was some virus checker going on, but you can get rid of those. Still, most of the problem just boils down to the duration of the delete on close being proportional

Re: unzip parallel, 3x faster than 7zip

2012-04-07 Thread Jay Norwood
On Saturday, 7 April 2012 at 11:41:41 UTC, Rainer Schuetze wrote: Maybe it is the trim command being executed on the sectors previously occupied by the file. No, perhaps I didn't make it clear that the rmdir slowness is only an issue on hard drives. I can unzip the 2GB archive in about

Re: unzip parallel, 3x faster than 7zip

2012-04-07 Thread Jay Norwood
On Saturday, 7 April 2012 at 17:08:33 UTC, Jay Norwood wrote: The mydefrag program uses the ntfs defrag api. There is an article at the following link showing how to access it to get the Logical Cluster Numbers on disk for a file. I suppose you could sort your file operations by start LCN,

Re: unzip parallel, 3x faster than 7zip

2012-04-06 Thread Sean Cavanaugh
On 4/5/2012 6:53 PM, Jay Norwood wrote: I'm curious why win7 is such a dog when removing directories. I see a lot of disk read activity going on which seems to dominate the delete time. This doesn't make any sense to me unless there is some file caching being triggered on files being

Re: unzip parallel, 3x faster than 7zip

2012-04-06 Thread Jay Norwood
On Friday, 6 April 2012 at 14:55:14 UTC, Sean Cavanaugh wrote: If you delete a directory containing several hundred thousand directories (each with 4-5 files inside, don't ask), you can see windows freeze for long periods (10+seconds) of time until it is finished, which affects everything up

Re: unzip parallel, 3x faster than 7zip

2012-04-06 Thread dennis luehring
Am 05.04.2012 19:04, schrieb Timon Gehr: On 04/05/2012 06:37 PM, dennis luehring wrote: Am 05.04.2012 16:04, schrieb Jay Norwood: I uploaded a parallel unzip here, and the main in the examples folder. Testing on my ssd drive, unzips a 2GB directory structure in 17.5 secs. 7zip took 55 secs

Re: unzip parallel, 3x faster than 7zip

2012-04-06 Thread dennis luehring
Am 06.04.2012 01:53, schrieb Jay Norwood: I'm curious why win7 is such a dog when removing directories. I see a lot of disk read activity going on which seems to dominate the delete time. try windows safe-mode (without network :} - your virus scanner is disabled), press F8 before windows

Re: unzip parallel, 3x faster than 7zip

2012-04-05 Thread Jay Norwood
On Thursday, 5 April 2012 at 14:04:57 UTC, Jay Norwood wrote: I uploaded a parallel unzip here, and the main in the examples folder. So, below is a demo of how to use the example app in windows, where I unzipped a 2GB directory structure from a 1GB zip file, tzip.zip. 02/18/2012 03:23 PM

Re: unzip parallel, 3x faster than 7zip

2012-04-05 Thread Jay Norwood
On Thursday, 5 April 2012 at 15:07:47 UTC, Jay Norwood wrote: so, a few comments about std.zip... I attempted to use it and found that its way of unzipping is a memory hog, keeping the full original and all the unzipped data in memory. It quickly ran out of memory on my test case.

Re: unzip parallel, 3x faster than 7zip

2012-04-05 Thread dennis luehring
Am 05.04.2012 16:04, schrieb Jay Norwood: I uploaded a parallel unzip here, and the main in the examples folder. Testing on my ssd drive, unzips a 2GB directory structure in 17.5 secs. 7zip took 55 secs on the same file. it makes no sense to benchmark different algorithm zip-7zip compare

Re: unzip parallel, 3x faster than 7zip

2012-04-05 Thread Timon Gehr
On 04/05/2012 06:37 PM, dennis luehring wrote: Am 05.04.2012 16:04, schrieb Jay Norwood: I uploaded a parallel unzip here, and the main in the examples folder. Testing on my ssd drive, unzips a 2GB directory structure in 17.5 secs. 7zip took 55 secs on the same file. it makes no sense to

Re: unzip parallel, 3x faster than 7zip

2012-04-05 Thread Jay Norwood
I think he is talking about 7zip the standalone software, not 7zip the compression algorithm. 7zip took 55 secs _on the same file_. Yes, that's right, both 7zip and this uzp program are using the same deflate standard format of zip for this test. It is the only expand format that is