More numbers... Same database, this time on a desktop, Windows Server 2008 R2, 2GB RAM, Inter Xeon 3050 2*13 GHz.
checkpoint_after |doc_buffer_size | avg. compaction time (in minutes) ==================================================================== 41943040 | 4194304 | 17:26 83886080 | 8388608 | 12:01 167772160 | 16777216 | 9:34 335544320 | 33554432 | 8:56 671088640 | 67108864 | 9:26 (*) 1342177280 | 134217728 | 10:20 2684354560 | 268435456 | 12:28 671088640 | 4194304 | 17:25 671088640 | 8388608 | 12:17 671088640 | 16777216 | 10:15 671088640 | 33554432 | 8:56 671088640 | 67108864 | 9:20 (*) 671088640 | 134217728 | 10:16 671088640 | 268435456 | 12:19 On Fri, Nov 1, 2013 at 5:44 PM, Boaz Citrin <[email protected]> wrote: > OK some numbers when running compaction of a database of 1.5M docs on a > Dell Latitude laptop with Windows 7 64b OS, 8G RAM, Intel Core i7-3720QM > CPU (2 * 2.6 GHz). > > checkpoint_after |doc_buffer_size | avg. compaction time (in minutes) > ====================================================== > 41943040 | 4194304 | 6:11 > 83886080 | 8388608 | 5:10 > 167772160 | 16777216 | 4:49 > 335544320 | 33554432 | 4:06 > 671088640 | 67108864 | 3:34 (*) > 1342177280 | 134217728 | 4:16 > 2684354560 | 268435456 | 5:19 > > 671088640 | 4194304 | 5:39 > 671088640 | 8388608 | 4:06 > 671088640 | 16777216 | 3:44 > 671088640 | 33554432 | 3:20 > 671088640 | 67108864 | 3:16 (*) > 671088640 | 134217728 | 3:44 > 671088640 | 268435456 | 4:28 > > As you can see in the starred lines, same parameters might give different > results, but I think the direction is clear; > In my situation buffer of 64m gives the fastest compaction, and we also > see that as expected the checkpoint size affects the compaction time as > well. > > Note that this is NOT the time that an average daily compaction takes as I > ran these consecutively, i.e. the database was initially not fragmented. > > > > On Sat, Oct 26, 2013 at 5:25 PM, Alexander Shorin <[email protected]>wrote: > >> Feel free to post here! I think others would be also interested in >> your experience and could share (I hope so) their own too. >> -- >> ,,,^..^,,, >> >> >> On Sat, Oct 26, 2013 at 7:05 PM, Boaz Citrin <[email protected]> wrote: >> > Hi Alexander, >> > >> > Yes, I have some numbers, do you want me to share here or somewhere >> else? >> > >> > Best, >> > >> > Boaz >> > >> > >> > On Mon, Oct 21, 2013 at 12:32 AM, Alexander Shorin <[email protected]> >> wrote: >> > >> >> Hi Boaz! >> >> >> >> On Fri, Oct 18, 2013 at 12:57 AM, Boaz Citrin <[email protected]> >> wrote: >> >> > Testing database compaction with various doc_buffer_size values I get >> >> > completely different results. >> >> > Documentation is fairly vague, so I wonder how to choose the right >> value; >> >> > What parameters affect this - HD buffer size, average doc size, >> >> > database size, fragmentation, etc... ?! >> >> > >> >> > Same goes for view compaction and keyvalue_buffer_size. >> >> >> >> These parameters does affect on what they are named: they defines >> >> buffer size for copying data from db/view file to the .compact one. >> >> What information you think is missed? >> >> >> >> > (For me the the compaction with default values was many times slower >> >> > than with the values that gave the faster compaction). >> >> >> >> All numbers have their cost: large buffers requires more memory while >> >> they reduces I/O operations and vice versa. Much likely, that default >> >> values wouldn't provide you high performance since they aimed to fit >> >> everyone, but that's why you may tweak them for your needs (: >> >> >> >> I believe, that it's possible to revise them, but first need to >> >> collect information in what environment which values are effective and >> >> which are not. Would you like to help us with that? >> >> >> >> >> >> -- >> >> ,,,^..^,,, >> >> >> > >
