(This is a very specialized experimental patch that may only work on
FreeBSD as it uses kqueue(). Furthermore, this patch is only intended
for advanced users who understand the significance of tightly coupling
qmail-queue to qmail-send.)
Some sites have problems with their peak queue injection rate
exceeding the rate at which qmail-send can service the 'todo'
queue. Classic symptoms of a sustained peak are large values in the
qmail-qstat output like this:
messages in queue but not yet preprocessed: 1092
On many, but not all Unix file systems a large 'todo' directory
adversely affects the performance of qmail exponentially and
permanently.
Some people resort to the big-todo patch as a way of ameliorating this
problem, but big-todo imposes a significant additional load that can
create more problems than it solves.
One solution is to get a faster system - especially one with faster
disk(s).
Another possible solution is this patch. What this patch does is
modify qmail-queue so that it does not exit until the queued file in
intd has been unlinked - this normally means that qmail-send has
pre-processed the queued file. In other words this patch tightly
couples the qmail-queue injection rate to the rate at which qmail-send
can process the 'todo' queue.
If you have control over the number of instances of qmail-queue, eg a
concurrency limit for a webmail application or a concurrency limit on
a script that injects a large amount of mail or even the normal
concurrency control that tpcserver offers, then you can eliminate the
risk of creating a large 'todo' directory.
Note well though, if qmail-send is not running, qmail-queue will
stall! While the code has a timeout provision, using it will almost
certainly create lots of duplicates. Besides, this patch is only
intended for well run, well monitored systems. It doesn't hurt to
restart qmail-send, just be aware that all injections will wait until
qmail-send restarts and drains 'todo'.
I wrote this patch for a set of systems that have wildly varying
injection rates and it does what I need - provide a resource
reservation mechanism - but as always, use of this patch is entirely
at your own risk.
Less brittle coupling will probably have to wait for zeroseek.
Regards.
diff -cN qmail-1.03/Makefile qmail-1.03-wfu/Makefile
*** qmail-1.03/Makefile Mon Jun 15 03:53:16 1998
--- qmail-1.03-wfu/Makefile Thu Aug 16 16:39:58 2001
***************
*** 1421,1431 ****
qmail-queue: \
load qmail-queue.o triggerpull.o fmtqfn.o now.o date822fmt.o \
datetime.a seek.a ndelay.a open.a sig.a alloc.a substdio.a error.a \
! str.a fs.a auto_qmail.o auto_split.o auto_uids.o
./load qmail-queue triggerpull.o fmtqfn.o now.o \
date822fmt.o datetime.a seek.a ndelay.a open.a sig.a \
alloc.a substdio.a error.a str.a fs.a auto_qmail.o \
! auto_split.o auto_uids.o
qmail-queue.0: \
qmail-queue.8
--- 1421,1436 ----
qmail-queue: \
load qmail-queue.o triggerpull.o fmtqfn.o now.o date822fmt.o \
datetime.a seek.a ndelay.a open.a sig.a alloc.a substdio.a error.a \
! str.a fs.a auto_qmail.o auto_split.o auto_uids.o \
! wait_for_unlink.o
./load qmail-queue triggerpull.o fmtqfn.o now.o \
date822fmt.o datetime.a seek.a ndelay.a open.a sig.a \
alloc.a substdio.a error.a str.a fs.a auto_qmail.o \
! auto_split.o auto_uids.o wait_for_unlink.o
!
! wait_for_unlink.o: \
! compile wait_for_unlink.c
! ./compile wait_for_unlink.c
qmail-queue.0: \
qmail-queue.8
diff -cN qmail-1.03/TARGETS qmail-1.03-wfu/TARGETS
*** qmail-1.03/TARGETS Mon Jun 15 03:53:16 1998
--- qmail-1.03-wfu/TARGETS Thu Aug 16 16:40:37 2001
***************
*** 40,45 ****
--- 40,46 ----
sig_bug.o
sig_misc.o
sig.a
+ wait_for_unlink.o
open_append.o
open_excl.o
open_read.o
diff -cN qmail-1.03/qmail-queue.c qmail-1.03-wfu/qmail-queue.c
*** qmail-1.03/qmail-queue.c Mon Jun 15 03:53:16 1998
--- qmail-1.03-wfu/qmail-queue.c Thu Aug 16 16:38:47 2001
***************
*** 249,254 ****
--- 249,256 ----
if (link(intdfn,todofn) == -1) die(66);
+ wfu_register(intdfd);
triggerpull();
+ wfu_wait(0);
die(0);
}
diff -cN qmail-1.03/wait_for_unlink.c qmail-1.03-wfu/wait_for_unlink.c
*** qmail-1.03/wait_for_unlink.c Wed Dec 31 16:00:00 1969
--- qmail-1.03-wfu/wait_for_unlink.c Thu Aug 16 16:38:16 2001
***************
*** 0 ****
--- 1,43 ----
+ #include <stdio.h>
+ #include <sys/time.h>
+ #include <sys/types.h>
+ #include <sys/event.h>
+
+ static int kfd = -1;
+
+ extern int
+ wfu_register(int fd)
+ {
+
+ struct kevent ev_in;
+
+ if ((kfd = kqueue()) == -1) return -1;
+
+ ev_in.ident = fd;
+ ev_in.flags = EV_ONESHOT | EV_ENABLE | EV_ADD | EV_CLEAR;
+ ev_in.filter = EVFILT_VNODE;
+ ev_in.fflags = NOTE_DELETE;
+ return kevent(kfd, &ev_in, 1,
+ NULL, 0,
+ NULL);
+ }
+
+
+ extern int
+ wfu_wait(int max_delay_seconds)
+ {
+
+ struct kevent ev_out;
+ struct timespec to;
+
+ if (kfd == -1) return -1;
+
+ if (max_delay_seconds > 0) {
+ to.tv_sec = max_delay_seconds;
+ to.tv_nsec = 0;
+ }
+
+ return kevent(kfd, NULL, 0,
+ &ev_out, 1,
+ (max_delay_seconds > 0) ? &to : NULL);
+ }
