Hello, On Sun, Feb 16, 2014 at 10:29:34AM -0800, Nam H Nguyen wrote: > Dear HAProxy Develop Team, > We have been using your product as our Load Balancer for a while and we are > very glad of what the product have brought us. However, we sometimes ran into > an issue that we suspect it might be because of a bug in a load balancer. > Here is the situation: > We are using HAProxy as a Load Balancer in front of our 2 Glassfish servers > which host multiple applications on them. The servers are running on Amazon > EC2 and are identical. All applications are all web services using JAX-RS > without any session states. Our web services receive POST messages from > different devices and the messages will be processed and stored in a > DATABASE. In some occasions, messages from one application were mixed with > different messages from other application and create garbage which could not > be understood by the web service and will be logged. It seems like a partial > of one message (with random length) is insert to random position of another > message. The reason we suspect there is a bug in HAProxy because we have used > a commercial product Riverbed Load Balancer before without any issue until we > switched to HAProxy. > We tried to look up on the Internet to see if any other people have the same > issue like us and it seems like some of them having the same issue. Here is > the source: > > http://www.coderanch.com/t/627744/Web-Services/java/glassfish-SOAP-requests > > > Here is also the current config file of HAProxy that we are using on our Load > Balancer: > > > global > log /dev/log local0 > log /dev/log local1 notice > maxconn 10000 > chroot /var/lib/haproxy > user haproxy > group haproxy > daemon > > defaults > log global > mode http > option httplog > option dontlognull > retries 3 > contimeout 5000 > clitimeout 50000 > srvtimeout 50000 > errorfile 400 /etc/haproxy/errors/400.http > errorfile 403 /etc/haproxy/errors/403.http > errorfile 408 /etc/haproxy/errors/408.http > errorfile 500 /etc/haproxy/errors/500.http > errorfile 502 /etc/haproxy/errors/502.http > errorfile 503 /etc/haproxy/errors/503.http > errorfile 504 /etc/haproxy/errors/504.http > > listen loadbalancer 0.0.0.0:80 > mode http > stats enable > stats uri /haproxy?stats > stats realm Strictly\ Private > stats auth admin:###### > balance roundrobin > # option httpclose > option forceclose > option forwardfor > server server1 10.0.0.121:8080 check > server server2 10.0.0.122:8080 check > > > > > We hope that the information is useful for you to improve the product and > also if you can point out any misconfiguration that we have done on our > HAProxy config file so that we can fix the issue.
I don't think it's a misconfiguration, but at the same time, the issue you describe looks impossible to me given the way the sockets are tied to buffers and to one session. The only case in history where such an issue was reported was with TCP splicing due to a kernel bug causing pages to be shared between multiple pipes. But here you're not even using splicing. What I suspect instead would be an issue in the content-length computation somewhere in the chain or with a chunked encoding. For example, I have some memories about glassfish causing issues in the past when fed with "connection: close" : it would disable HTTP/1.1 hence disable chunked encoding, sometimes resulting in truncated responses in case of any network error. Please could you send the output of "haproxy -vv" so that we can see all the build options in case where something rings a bell ? I really think that a network capture of both sides of haproxy will be required to troubleshoot this. Best regards, Willy
