Guys, I found a workaround which seems to be working quit ewell at the moment. For some reason the kernel seems to ignore the max TCP window size when GSO is enabled on the interface, resulting in hundreds of kB in flight which take ages to recover in case of losses => haproxy sees nothing move and finally times out. Disabling GSO on that interface completely fixed the issue, now the socket's send queues are reasonable and match the configuration and I've not seen a timeout for the last hour. There were always a few per hour previously that I always attributed to the clients!
So I think it's really fixed now. Cheers, Willy