On Thursday 25 October 2001 05:07, Loren Frank wrote:
> Hi all. First off, thanks to those who responded to my previous post.
> Turning on IRQ and DMA for the hard drive solves the problem.
>
> Now I have yet another slightly off topic disk question. I'm collecting
> data from somewhere between 12 and 24 sources and I in the data file
> produced by the collection software, the sources are all interleaved. I
> need to create separate files, one for each source, and I'm wondering
> if anyone has an idea of how much performance hit there would be for
> writing them all out simultaneously rather than one at a time. The
> original data file could be as large as a few GBytes, and I don't
> relish the idea of going through it 24 times, one for each source.
This is very similar to what we're doing around the linux-audio-dev
mailing list; hard disk recording and editing of multitrack audio.
There are a few things to keep in mind here; most importantly:
* Normal file/disk system implementations aren't designed to
handle multiple simultaneous high bandwidth streams.
* The delayed write and read-ahead features built into nearly
every modern OS in existence generally cause trouble rather
than help performance.
* Hard drive seek time is *dead* time. *DON'T* write in the
order of tens of kB at a time, like some clueless Windows
audio software companies used to suggest. (Just think about
it; 32 kB blocks for 16 192 kB/s streams ==> 6 blocks per
stream and second ==> 96 seeks/second (in a perfect world).
You'll need a <10 ms avg. seek drive to even get away with
the *seeking* - and you still have no time for actually
accessing the disk...)
* Hard drives have all heads mounted on *one* seek motor, and
they generally use only one head at a time.
That is, for maximum throughput, set up a buffer of in the order of
*hundreds* of kB for each channel. Perform all writes sequentially, one
channel at a time from a single thread per disk. (That is, only one "disk
writer" thread, unless you're using multiple drives.) Never write less
than a few hundred kB from each buffer - if there's no buffer with enough
data, just wait until there is. (This rule is required to prevent buffer
level and/or CPU timesharing fluctuations from stressing the disk writer
thread into doing stupid things.) Don't let the buffers fill up too much!
You need the margin to make sure you don't lose data due to buffer
overruns. (Remember that we're dealing with a *very* non-deterministic
subsystem here! When you block on a write, expect to stay blocked for up
to a full "write one block for each channel" cycle in the worst case -
perhaps even longer, if you're running close to the bandwidth limits.)
With this kind of setup, it's possible to use 50% or more of the system's
sustained disk transfer rate for multichannel streaming, with 50+ streams
on Your Average EIDE PC. The current 7500 and 10000 rpm DMA/100 drives
should be capable of a lot more than that, but do keep in mind that the
seek times aren't that much better! It's all about proper use of
buffering, to minimize the number of seeks per cycle over all channels.
In short, streaming multiple high bandwidth channels to/from a single
drive is indeed doable, although not entirely trivial. The guidelines
above, and some common sense, should be enough to get it working, though.
//David Olofson --- Programmer, Reologica Instruments AB
.- M A I A -------------------------------------------------.
| Multimedia Application Integration Architecture |
| A Free/Open Source Plugin API for Professional Multimedia |
`----------------------------> http://www.linuxdj.com/maia -'
.- David Olofson -------------------------------------------.
| Audio Hacker - Open Source Advocate - Singer - Songwriter |
`-------------------------------------> http://olofson.net -'
-- [rtl] ---
To unsubscribe:
echo "unsubscribe rtl" | mail [EMAIL PROTECTED] OR
echo "unsubscribe rtl <Your_email>" | mail [EMAIL PROTECTED]
--
For more information on Real-Time Linux see:
http://www.rtlinux.org/
