Well, the kernel memory allocator itself doesn't use swap, so it is more likely a runaway user program (?) and not the kernel. If tmpfs is being used it could be something filling up the tmpfs mount, including possibly a process with a descriptor open on an unlinked file. That's all I can think of.
Under normal operation idle processes will eventually wind up getting paged out since there is no sense tying up ram to hold them. e.g. things like getty and such, but these normally do not amount to very much use. -Matt