There is really no need to set ulimit-n manually, haproxy will do that
for you (as per the docs "it is recommended not to use this option").
Also, configuring maxconn correctly is the single most important
configuration in haproxy, and will affect the performance as well as
cause crashes if the OOM killer shots down a miss-configured haproxy
process.
You should've seen the OOM kills in your dmesg though.
Regards,
Lukas