On 03/08/11 05:43, Bob Clary wrote:
On 3/8/11 1:09 AM, Ian Hobson wrote:
After compacting, the database will have a given size on disk. Would it
be possible to test, and compact if this grew by (say) 15%?
Wayne,
You say that your database size is 0.6 TB. What is the change in size
during the day? What is the change in size after the compaction? If your
database is not increasing appreciably in size during the day and if the
compacted database size is not appreciably smaller than the
pre-compaction size, I don't think you are gaining much by compacting
once per day. In fact, you are taking a significant performance hit if
your compaction is running for 10 hours every day.
Perhaps a simple change in compaction schedule from once per day to once
per N days will help in the short term. Similarly to Robert's
suggestion, I keep track of the initial size of the database as well as
the initial sizes of each of the views and compact them whenever they
double in size.
Of course you will need to tailor the trigger point so as to not reach a
point where you do not have sufficient disk space to complete the
compaction.
Until Bob Dionne's patch is released I think that is the best you can
achieve.
Unless compaction performance is significantly improved, you also need
to consider the case where once your database grows to a sufficiently
large size, your database will reach a state where it is always being
compacted, i.e., it will take so long to complete compaction that it
will need to be compacted again immediately after it finishes.
/bc
Bob,
We backed off on how often we were triggering compaction when we noticed
it taking 10 hours to do. Once I'm doing sharding, I am going to use a
growth strategy such as Iam Hobson suggests or the one you mention,
though: A database that has not grown (much) is one that does not need
compaction.
Today I started splitting the 0.6TB up into a few hundred individual
databases. Many of them get little traffic, so this will help
significantly once the patch is available and I can compact only when
there's space to be gained. It will help even when using growth as a
trigger.
Thanks,
Wayne Conrad
