John, I will throw in my 2 cents:
There are a number of open-sourced offerings done by well known companies in this area and I have tried a few of them. Generally my opinion is that they are a little restrictive. We are using the HAProxy service on each slave as you described. We have attempted to leverage marathon API and then the event-notification feature to make HAProxy config more reactive to the changes. The even-notification was too noisy so we moved away from it to simply making per-minute calls to marathon. We have now moved away from marathon to singularity which is another story. For all I have read, I am not keen on putting any additional load on zookeeper. HAProxy is rock solid and we have not looked back on the decision so far. Your point of 64K ports is perfectly valid. We are far away from that limitation today. Your 2nd point of HAProxy terminating connections after 50K ms is more interesting. We may run into that issue. Here I would be looking for a possible HAProxy configuration setting. Zia
