On 05/02/2015 19:22, Paul Eggert wrote:
On 02/05/2015 09:04 AM, Mark Lehrer wrote:
I haven't yet done a full blktrace analysis yet, but the 250MB/sec
speed seems to match other tests I have done with a 512 byte block
size. Is it possible to do the equivalent of dd's "obs" option, to
increase the output block size to 4k, 64k, or even 1M?
Sure, the -b option does that. The default is 20 (i.e., 20 x 512 = 10
KiB).
Which happens not to be a multiple of 4Kio.
has anyone tried to make a multi-threaded version?
Not as far as I know. It's not clear that going multithreaded would
be worth the hassle.
I would agree, but given the typical behavior of correctly
implemented file system flush logic, it might pay to somehow
overlap the closing of extracted regular files with the
extraction of subsequent files (because close(fd) must imply
fdflush(fd) which must wait for disk I/O thus preventing
efficient coalescing of metadata writes to any media, even
SSDs). Additionally, if an on-access "virus scanner" is
installed, close(fd) may also trigger a wait for the
extracted file being checked by the scanner process. A
simple form could be to "batch together" the close() calls
in bundles of e.g. a few dozen or a few hundred. A more
advanced form would delegate the close() calls to a group
of helper threads which are awoken for each batch of file
handles, and which pick up queued close() calls one at a
time from a shared queue (because the time to complete each
call will vary and having multiple close() calls in parallel
gives kernels the best chance to coalesce meta-data writes).
During archive creation, two optimizations would help with
disk cache interactions:
1. Do open(O_RD) calls ahead of time in background threads
so the file handles will usually be ready for read() calls
as soon as the end of the previous file is reached. This
should be a speedup when taring up many small files, such
as a large source code tarball.
2. In --sparse mode, use a large buffer to only do one pass
over each file. With current tar formats this means
building up an in-memory representation of the non-zero
file blocks and their locations, until the buffer is
filled, then output the chunk using the special tar records
already defined for a huge file that has been split into
multiple tar entries for other reasons. This should be a
speedup when taring up highly sparse larger-than-memory
files such as virtual machine disk images. It should also
protect against race conditions caused by blocks changing
from zero to non-zero between passes, and/or the file length
changing during file reading (such things can happen when
dumping a live file system without the benefit of a cow
disk snapshot mechanism).
Enjoy
Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. http://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
