> I just want to tell you that the most recent version of
> dump/restore (version 0.4b23) plus a tiny patch (see below)
> enables you to pipe the output of dump directly into
> cdrecord. This includes compression and splitting into
> chunks which fit onto a CD. The compression is quite effective,
> my 5.8 Gb root fs fits onto 2 CDs (0.65 Gb each) and
> my /usr/local fs (over 9Gb) fits onto 7 CDs (0.7 Gb each).
A couple of weeks ago, somebody expressed a desire for a way to vary
the compression level dynamically, based on the CPU speed and the
actual speed of the drive.
I've worked up an initial, somewhat crude, but apparently effective
modification to the patched version of 0.4b23 which extends its
compression support with this sort of dynamic compression-level
adjustment. A context diff may be found in:
http://www.radagast.org/~dplatt/unix/dump-0.4b23-dynamic.diff
This modification adds some code to the slave-writer logic in
dump/tape.c which can vary the compression level. The method is
a simple one: the program tries to keep the compression level tuned
to the point at which the compressed data is being written to the
output device "just in time".
The code measures the amount of time which each write() call takes.
If it's more than a certain amount (by default, two clock ticks), the
code "believes" that the writes are being blocked because the writer
FIFO buffer is full, and it gradually increases the data-compression
level. When a write() call takes less than the time threshold, the
code "believes" that the writer FIFO is falling below the 100%-full
level, and reduces the compression level (so as to use less CPU time,
write the data sooner, and perhaps write data which is less
efficiently compressed and thus fill up more of the FIFO). In cases
where the compression is falling fairly far behind (e.g. a fast CD-R
burner) the compression will drop back all the way to "level 0", or no
compression at all.
This method (measuring the write completion time, to try to gauge the
FIFO's fill level) is not at all elegant. It's quite indirect and
imprecise. Even if the compression process is keeping up with the
burner, it's likely that some of the slaves' write() calls will
complete immediately - any time the burning software does a big read()
from standard input, several of the slave write() calls are likely to
complete within a tick or two, and the compression-adjustment
algorithm is likely to lower the compression level unnecessarily. A
better method would be one in which the dump routine had access to the
current "fill level" in the burner-software FIFO - this would require
much closer integration between dump and cdrecord.
Three new parameters can be used to tune the heuristic:
-I # - compression increment, specified in percent.
Default is zero (no dynamic increment).
-D # - compression decrement, specified in percent.
Default is zero (no dynamic decrement).
-l # - threshold tick count. Default is 2 ticks.
Some amount of experimentation and tuning is likely to be required on
any given system, in order to give the highest possible compression
level without starving the output FIFO.
Initial tests on my home system (K6-2-400, Plextor writer running at
4x) show that the logic seems to work.
Without dynamic level adjustment, and running at a compression level
of 6, my CPU can't keep up with the burner. The cdrecord FIFO level
repeatedly drops to 0, and the Plextor burner stops burning. If it
weren't for the use of BURN-PROOF, the backup would have made a
useles coaster and failed. With BURN-PROOF, the backup is usable, and
achieves a compression level of about 2.5:1 on my /home filesystem.
Without dynamic level adjustment, running at a compression level of 1,
my CPU keeps up with the burner (average CPU load is somewhere around
75% I think). BURN-PROOF is not necessary, and I got a compression of
about 2.35:1.
With dynamic level adjustment (-I 5 -D 5) the compression level drops
down quite rapidly, jitters between 1 and 0 every couple of seconds,
and occasionally pops up to 3 or 4 for a while. The cdrecord FIFO
level sits at 100% most of the time (I never saw it drop below about
94%) and the Plextor never stops burning. The resulting backup
verifies correctly, and has a compression level of 1.85:1.
I suspect that tuning the dynamic level adjustment parameters will
produce better results. It may be better to have the -I parameter
equal to several times the -D parameter, so that the occasional (and
expected) "quick write completion" doesn't decrease the compression
level unnecessarily. Something like -I 10 -D 3, or -I 50 -D 5 might
do the job. Setting the "quick write" threshold to 1 tick (-l 1)
might also help.
I tried a backup on my Celeron 400 at work, with a 2x burner, with
-z6 -I10 -D1 -l1. The dynamic algorithm fairly quickly nudged the
compression level up to 9 for the duration of the backup... it dropped
back down to 8 for a few seconds at one point and then went back up to
9. The CPU was about 60% busy during the backup.
Clearly, there are a bunch of approaches and tradeoffs. The best
compression will be achieved with -z9, without dynamic adjustment, and
with a slow burn speed (or with BURN-PROOF or similar to avoid
coasterizing). Using lower -z compression levels would significantly
reduce the CPU impact of running the backup, and would reduce the risk
of coasters if the system were subject to sudden CPU or disk I/O
stresses (e.g. other users running jobs during the backup).
Using dynamic compression adjustment could be an attractive option for
people who don't have burners equipped with BURN-PROOF. It could
allow the burner to run at the highest speed which "usually works
reliably, with a reasonable compression level", and would reduce the
compression level as necessary in order to keep the burner from
starving during times of CPU or disk stress.
So... if you're interested, download the patch from the URL above and
try it out. I'd appreciate hearing your results, reactions,
suggestions for improvements or alternate algorithms, etc.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
