Hello again,
It turns out that it's not a FAT issue, but that the same problem occurs
on ext3 systems as well. I've written a small program to test the delays
between writes, and the results are not very encouraging. Specially when
the disk gets full (it always does, doesn't it?).
In my opinion, this should concert anyone who want to use a Linux box
for storing a data stream (audio, video, whatever).
I've attached the source of the program I used. Basically, it loops on
writing 32kB chunks of data to a file, creating a list of number telling
how much time (in microseconds) elapsed since the last loop (to stdout).
There are two modes of testing: One is to let it write as fast as
possible, and the second is to put delays between writes, which
simulates waiting for incoming data. If there is enough room, the
program will write slightly less than 2 GB (guess why?).
Since Linux is a multitasking system, the results are not exactly
repeatable. But the general impression is that writing to FAT or on
ext3, on my laptop or on my desktop, they all behave more or less the same.
First test regards full-speed write. Data was simply written as fast as
possible. For a non-full partition, the write operation dwelled
typically 5.5 ms, with occasional bursts of 0.7-0.9 *seconds* delay on
the write operation. When the partition gets full, things get even
nastier. Several seconds of blocking was observed. 5 seconds, and up to
14 seconds delay typically appeared a few times for a 2 GB writing session.
Then I added a short sleep in the loop, in order to simulate data
written at ~ 3MB/sec (which is reasonable for video capturing). This is
far below the disk's physical capicity. The disk LED showed occasional
flushes.
Results: For a non-full partition, occasional peaks of up to 60ms were
observed, which is something one can live with, probably. At 3 MB/s this
means 180 kB stuck in the buffer. But when the partition started to get
full, peaks of 0.2-0.3 seconds started to appear. The latter means 900
kB waiting to go out, and this maybe explains why I originally had problems.
If you want to see how your system behaves, just compile the attached
code and go:
./writefat output-junk-data-file > listfile
The list of loop timings will be in listfile. Use your favourite number
cruncher to view graphs. (The program's name is due to "historic
reasons"...) If you want to test the slower writing speeds, check the
typical delay in the listfile, or see how fast the output file grows.
The sleep period defined in the program itself is not reliable, since
the operating system may not be able to sleep for too short periods.
And finally: Does an RAM FIFO help? Surprisingly, the answer is no. I
did the following:
mknod mypipe p
mbuffer -i mypipe -o /fatfs/output-file &
./writefat mypipe > listfile
and was quite surprised to find delays of 0.2 sec. BTW, mbuffer seems to
force the data to be flushed to disk much more often. The disk LED
showed that writes occured all the time, unlike the direct write to
file, in which flushes occured occasionally. And "mypipe" and "logfile"
are on an ext3, while outfile-file is on FAT.
Insights, anybody?
Eli
--
Web: http://www.billauer.co.il
#include <stdio.h>
#include <sys/time.h>
#include <unistd.h>
char junkdata[32768];
int main(int argc, char **argv) {
FILE *f;
struct timeval now, sleeptime;
struct timezone junk;
long prev_sec, prev_usec, deltat;
int i;
if (argc!=2) {
fprintf(stderr, "Usage: %s output-filename\n", argv[0]);
return 1;
}
f = fopen (argv[1], "wb");
if (f==NULL) {
fprintf(stderr, "Failed to open output file %s\n", argv[1]);
return 1;
}
if (gettimeofday(&now, &junk) != 0) {
fprintf(stderr, "gettimeofday() failed!\n");
return 1;
}
prev_sec = now.tv_sec; prev_usec = now.tv_usec;
for (i=0; i<2047*32; i++) { // Almost 2 GB
// Time to sleep between writes. Check this program's output values
// to see what you actually got as typical values. They may be
// significantly longer than requested due to context switch overhead
// when the desired time is very short.
sleeptime.tv_sec = 0;
sleeptime.tv_usec = 100; // Yeah, right. The real value will be much longer
// Next line should be commented out for full-speed
// select(0, NULL, NULL, NULL, &sleeptime);
if (fwrite (junkdata, 1, sizeof(junkdata), f) != sizeof(junkdata)) {
fprintf(stderr, "Data write failed!\n");
return 1;
}
if (gettimeofday(&now, &junk) != 0) {
fprintf(stderr, "gettimeofday() failed!\n");
return 1;
}
deltat = now.tv_sec - prev_sec;
deltat = deltat * 1000000;
deltat += now.tv_usec - prev_usec;
prev_sec = now.tv_sec; prev_usec = now.tv_usec;
printf("%ld\n", deltat);
}
fclose(f);
return 0;
}
