On Tue, Jan 18, 2011 at 1:35 PM, Michael Meeks <[email protected]> wrote: > On Wed, 2011-01-12 at 10:47 -0600, Norbert Thiebaud wrote: >> indeed it is a dog. it took 15h20. >> conservatively at least 13 hours of it were in helcontent2, mostly I/O >> bound creating temp files and copying things around over and over. [..] > > A huge win. Personally, I find it hard to believe that the problem is > not I/O related
Well as I said above, it _IS_ I/O related, no doubt about that. > - ie. to do something -that- slow, a file-system has to > fsync (ie. physically move the disk heads) I imagine, I don't think much > else can explain it. Out of interest how fast is: > > mkdir /tmp/small ; time for (( a=1; a<=10000; a++ )) do touch /tmp/small/$a; > rm /tmp/small/$a; done > > for you ? I get 22 to 26 seconds (on my two test machines), one has > an SSD, the other a hard-disk, but you can see that neither write anything > to the disk in that time. real 0m30.254s user 0m5.720s sys 0m25.907s on my mac (note: doing this in a ram_disk give about the same result) real 0m20.760s user 0m2.974s sys 0m18.205s on my linux but that does not measure much since mkdir /tmp/small ; time for (( a=1; a<=10000; a++ )) do touch /tmp/small/$a; done real 0m17.662s user 0m3.012s sys 0m15.115s time cp -r /tmp/small /tmp/small2 real 0m2.508s user 0m0.119s sys 0m2.256s n_th@tpamac ~ $time rm -fr /tmp/small2 real 0m1.000s user 0m0.014s sys 0m0.808s So the test you give is more about measuring bash and fork speed than anything else > > So - I'm simply surprised that a ramdisk makes much difference. > > I was suspicious of the database / lucene building piece - which may > well do fsyncs, and write a journal to try to ensure data integrity, I > can believe that is horribly slow. That is a distinct possibility. but the speed-up is there even for the part that are not running lucene. (based on the cpu-load - io/load monitoring while building on regular disk or on ramdisk) for info time for (( a=1; a<=1000; a++ )) do touch /Volumes/ccache_ramdisk/xx$a; sync; rm /Volumes/ccache_ramdisk/xx$a; sync; done real 6m37.891s user 0m1.448s sys 6m33.465s note: is 1,000 not 10,000 and it is the same order of magnitude when using 'real' disk Note: I'm building is -P6 -- -P6, that may also have an impact. some I/O pattern involved may dog badly in a heavy multi-threaded/multi-processed environment... (yeah, yeah... on paper that sound crazy high, but in practice 6/6 give me the best result on this machine) Norbert > >> Note: that if you do not have enough memory to do that, you could at >> least move the TMPDIR do a ramdisk. you don't need a big one (not sure >> exactly how much, but I bet 500M should probably do) >> there are some step in helpcontent2 that do a LOT of creation/delete on TMP. > > Right; I guess it might be nice to have an 'strace -f -ttt' log (or Mac > equivalent) of a few minutes of the build - to see what is going on. > > Thanks ! > > Michael. > > -- > [email protected] <><, Pseudo Engineer, itinerant idiot > > _______________________________________________ LibreOffice mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/libreoffice
