Just to say that I've been having the same problem with a Soekris board since about 4.4. I haven't figured out what's going on, but strangely the problem is getting better with time (i.e. the rate at which mbufs are allocated decreases). I *think* that it was fine in 4.3 (though I never run the machine for any length of time with that kernel), so you could try that if you want to investigate.
I haven't been able to establish a correlation between allocated mbufs and (network) load either. The "solution" for me so far has been to keep a watchful eye and reboot the machine once too much memory is used, combined with a watchdog and monit to reboot the machine automatically if it becomes unresponsive. Lars