Could this massive slowdown from 2.9 to 3.9Gb be due to the 3Gb RAM limit on 32-bit systems? Given that 32-bit systems are limited to 4Gb RAM and the OS reserves to top 1Gb, this performance hit could be due to large amounts of paging.
2008/4/29 Gwern Branwen <[EMAIL PROTECTED]>: > On 2008.04.27 22:49:53 -0700, Jason Dagit <[EMAIL PROTECTED]> scribbled 12K > characters: > > > > Please do not apply this patch to darcs, I'm sending it in for testing > > purposes. > > > > Gwern, does this this work on your 64bit machine or files over the 4GB > > limit? > > > > Thanks, > > Jason > > > > Sun Apr 27 19:49:21 PDT 2008 Jason Dagit <[EMAIL PROTECTED]> > > * use mmap for readFileLinesPSetc and fix mmap signature > > [ghc] src/Exec.o > [ghc] src/FastPackedString.o > [ghc] src/OldFastPackedString.o > > src/FastPackedString.hs:596:7: > Not in scope: type constructor or class `CSize' > > You forgot the import I told you about: > > hunk ./src/FastPackedString.hs 104 > -import Foreign.C.Types ( CInt, ) > +import Foreign.C.Types ( CInt, CSize ) > > --- > > Now, that aside, there are some interesting performance characteristics: > > [EMAIL PROTECTED]:2581~/foo>time whatsnew -s [11:51AM] > A ./bigtempfile > darcs whatsnew -s 0.99s user 0.84s system 99% cpu 1.834 total > [EMAIL PROTECTED]:2581~/foo>duh bigtempfile [11:52AM] > 2.9G bigtempfile > 2.9G total > [EMAIL PROTECTED]:2582~/foo>rm bigtempfile [11:52AM] > [EMAIL PROTECTED]:2583~/foo>time head -c 4079869184 /dev/zero > bigtempfile > [11:52AM] > head -c 4079869184 /dev/zero > bigtempfile 0.57s user 15.19s system 11% cpu > 2:15.27 total > [EMAIL PROTECTED]:2584~/foo>time whatsnew -s [11:54AM] > A ./bigtempfile > darcs whatsnew -s 1.74s user 3.43s system 3% cpu 2:13.51 total > [EMAIL PROTECTED]:2586~/foo>time whatsnew -s [11:58AM] > A ./bigtempfile > darcs whatsnew -s 1.84s user 3.48s system 3% cpu 2:46.19 total > [EMAIL PROTECTED]:2589~/foo>time whatsnew -s [12:02PM] > A ./bigtempfile > darcs whatsnew -s 1.90s user 3.12s system 4% cpu 2:01.31 total > [EMAIL PROTECTED]:2590~/foo>rm bigtempfile [12:05PM] > [EMAIL PROTECTED]:2591~/foo>time head -c 3079869184 /dev/zero > bigtempfile > && time whatsnew -s [12:05PM] > head -c 3079869184 /dev/zero > bigtempfile 0.42s user 11.21s system 11% cpu > 1:37.89 total > A ./bigtempfile > darcs whatsnew -s 0.99s user 0.80s system 31% cpu 5.596 total > [EMAIL PROTECTED]:2592~/foo>rm bigtempfile&& time head -c 5079869184 > /dev/zero > bigtempfile && time whatsnew -s [12:07PM] > head -c 5079869184 /dev/zero > bigtempfile 0.59s user 19.83s system 16% cpu > 2:07.12 total > A ./bigtempfile > darcs whatsnew -s 2.12s user 4.37s system 2% cpu 4:11.74 total > [EMAIL PROTECTED]:2593~/foo>rm bigtempfile&& time head -c 6079869184 > /dev/zero > bigtempfile && time whatsnew -s [12:15PM] > head -c 6079869184 /dev/zero > bigtempfile 0.78s user 22.74s system 12% cpu > 3:08.20 total > A ./bigtempfile > darcs whatsnew -s 2.33s user 5.08s system 2% cpu 5:01.04 total > [EMAIL PROTECTED]:2596~/foo>rm bigtempfile&& time head -c 1079869184 > /dev/zero > bigtempfile && time whatsnew -s [12:23PM] > head -c 1079869184 /dev/zero > bigtempfile 0.14s user 3.33s system 13% cpu > 26.457 total > A ./bigtempfile > darcs whatsnew -s 0.28s user 0.44s system 7% cpu 9.879 total > > In case you don't feel like looking through this entire list, the summary is > this: a 1.1G file takes 10 seconds. A 5.7GB file takes 300 seconds. So a 6x > file increase gives a 30x time increase (and memory usage is just as bad or > even worse, though I didn't track it). > > Or here's another example: > > [EMAIL PROTECTED]:2601~/foo>rm bigtempfile&& time head -c 079869184 > /dev/zero > bigtempfile && time whatsnew -s && duh bigtempfile; rm > bigtempfile&& time head -c 979869184 /dev/zero > bigtempfile && time whatsnew > -s && duh bigtempfile; rm bigtempfile&& time head -c 1079869184 /dev/zero > > bigtempfile && time whatsnew -s && duh bigtempfile; rm bigtempfile&& time > head -c 2079869184 /dev/zero > bigtempfile && time whatsnew -s && duh > bigtempfile; rm bigtempfile&& time head -c 3079869184 /dev/zero > bigtempfile > && time whatsnew -s && duh bigtempfile; rm bigtempfile&& time head -c > 4079869184 /dev/zero > bigtempfile && time whatsnew -s && duh bigtempfile > head -c 079869184 /dev/zero > bigtempfile 0.01s user 0.24s system 90% cpu > 0.277 total > A ./bigtempfile > darcs whatsnew -s 0.07s user 0.03s system 5% cpu 1.702 total > 77M bigtempfile > 77M total > head -c 979869184 /dev/zero > bigtempfile 0.17s user 3.19s system 25% cpu > 13.127 total > A ./bigtempfile > darcs whatsnew -s 0.36s user 0.25s system 99% cpu 0.610 total > 936M bigtempfile > 936M total > head -c 1079869184 /dev/zero > bigtempfile 0.12s user 3.40s system 14% cpu > 25.045 total > A ./bigtempfile > darcs whatsnew -s 0.43s user 0.27s system 99% cpu 0.702 total > 1.1G bigtempfile > 1.1G total > head -c 2079869184 /dev/zero > bigtempfile 0.31s user 7.50s system 12% cpu > 1:01.27 total > A ./bigtempfile > darcs whatsnew -s 0.69s user 0.50s system 99% cpu 1.192 total > 2.0G bigtempfile > 2.0G total > head -c 3079869184 /dev/zero > bigtempfile 0.41s user 11.40s system 12% cpu > 1:34.22 total > A ./bigtempfile > darcs whatsnew -s 1.07s user 0.71s system 99% cpu 1.781 total > 2.9G bigtempfile > 2.9G total > head -c 4079869184 /dev/zero > bigtempfile 0.57s user 15.20s system 11% cpu > 2:14.94 total > A ./bigtempfile > darcs whatsnew -s 1.72s user 3.42s system 2% cpu 2:56.71 total > 3.9G bigtempfile > 3.9G total > > Notice how it blows up between 2.9G and 3.9G: a single extra gigabyte, and > our whatsnew -s time goes from 1.8 seconds to 174 seconds. > > I tried it with profiling, and it claims the majority of the time is being > used by linesPS. get_unrecorded calls smart_diff which calls gen_diff, which > begat diff_files, and in the fullness of time, diff_files summoned get_text, > which duly appointed linesPS to eat 100% of CPU time... So it's not clear to > me why >3 gig files blow up. > > --- > > Testing-wise, this patch looks good to me - I didn't see any segfaults on > large files like with my own attempts, and make test is clean as usual with > the exception of either_dependency.sh and whatsnew.pl. I think > either_dependency.sh usually fails, but I don't know about whatsnew.pl. > > --- > > Anyway, if the whatsnew.pl issue gets cleared up and a cleaner patch written > (I think usage of mmap is supposed to be conditional on Autoconf.lhs), I > think this change would be worth applying. It's an enormous speedup on small > to large files, and it isn't *that* much worse for >3gig files (which folks > on 32-bit systems can't even use in the first place, as I understand it). > > Future work might be to change FPS.hs's 'mmap' call to just call readFilePS > on too-large files (given that it already does something other than mmap on > too-small files). > > -- > gwern > Terrorism CMS 1080H Choe Firewalls Lander 669 Zen HF STEP > > _______________________________________________ > darcs-users mailing list > [email protected] > http://lists.osuosl.org/mailman/listinfo/darcs-users > > _______________________________________________ darcs-users mailing list [email protected] http://lists.osuosl.org/mailman/listinfo/darcs-users
