I just ran siege from the internal network to haproxy first; It would seem that the issue doesn't happen here (earlier tests were to the external IP on the BSD firewall, this one to the 10.x.x.x interface), so it might be that the BSD firewall is causing issues here?
Hmm, I just realized that since this is from the internal network, I hardcoded the server in /etc/hosts to one haproxy frontend. For completeness, I also ran the check on the other haproxy which seems to be working fine as well.
