Upon further investigation, the following has been discovered, but still no real resolution:
The backend server doesn't matter, we moved a large .JS file to another server and the exact same issue occurred. Since the issue didn't occur when HAProxy was in TCP mode, we figured this would be the case. When downloading the file from a web browser on the specific device in question it failed everytime. On the same device if we used WGET, it would fail randomly, but often it automatically retried and finished downloading successfully, ie: WGET failed with connection closed at byte 342424. Retrying. 206 Partial Content response, downloaded remaining 18209 bytes. When we disabled mod_deflate in Apache on the backend server, the issue resolved itself immediately. In Apache if I set: DeflateBufferSize 1000000 (JS file is about 363K), then the error changes from a chunked error to ERR_CONTENT_LENGTH_MISMATCH. We attempted an HAProxy developer trace, but I'm not sure I got all the data, or if there was other requests mixed up in it: https://drive.google.com/file/d/1GUZqjkl361O2kSejPI4jv-ioB_8VjBtq/view?usp=share_link Any further assistance would be greatly appreciated. Thanks. On Sat, May 13, 2023 at 1:04 PM Mike Benoit <mike.ben...@gmail.com> wrote: > > Also just in case, are you aware of a previous version that did not > > exhibit this behavior ? > > I don't recall an exact version. Since the problem was extremely > intermittent (ie: once or twice per day) until just recently, when we > finally could replicate it consistently. If I had to guess it started > occurring after a major Ubuntu upgrade, ie: 18.04 to 20.04. > > I'm not 100% sure on the exact log line that matches the packet capture, > but here are several all from around that time (same IP address). The > client hasn't been able to download this one file successfully every: > > May 11 11:03:51 gateway haproxy[249868]: 188.29.25.76:1769 > [11/May/2023:11:03:51.229] http-in backend_http/mattermost - 0/0/0/1/3 200 > 99492 - - ---- 7/3/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:04:06 gateway haproxy[249868]: 188.29.25.76:6021 > [11/May/2023:11:04:06.321] http-in backend_http/mattermost - 0/0/0/1/3 200 > 99492 - - ---- 6/1/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:04:41 gateway haproxy[249868]: 188.29.25.76:29182 > [11/May/2023:11:04:41.198] http-in backend_http/mattermost - 0/0/1/1/3 200 > 99492 - - ---- 11/6/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:05:12 gateway haproxy[249868]: 188.29.25.76:20183 > [11/May/2023:11:05:12.736] http-in backend_http/mattermost - 0/0/1/1/4 200 > 99492 - - ---- 7/4/1/1/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:06:06 gateway haproxy[249868]: 188.29.25.76:1766 > [11/May/2023:11:06:06.295] http-in backend_http/mattermost - 0/0/2/1/5 200 > 99492 - - ---- 8/3/1/1/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:08:07 gateway haproxy[249887]: 188.29.25.76:1780 > [11/May/2023:11:08:07.179] http-in backend_http/mattermost - 0/0/0/1/25 200 > 99492 - - ---- 15/3/2/2/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:09:06 gateway haproxy[249887]: 188.29.25.76:5455 > [11/May/2023:11:09:06.434] http-in backend_http/mattermost - 0/0/10/1/13 > 200 99492 - - ---- 21/5/0/0/0 0/0 "GET > /static/5280.1241ee858d210c2d16c9.css HTTP/1.1" > May 11 11:16:33 gateway haproxy[249887]: 188.29.25.76:1783 > [11/May/2023:11:16:33.258] http-in backend_http/mattermost - 0/0/0/1/3 200 > 99492 - - ---- 15/1/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:16:41 gateway haproxy[249887]: 188.29.25.76:1784 > [11/May/2023:11:16:41.777] http-in backend_http/mattermost - 0/0/0/1/3 200 > 99492 - - ---- 14/1/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:17:07 gateway haproxy[249887]: 188.29.25.76:1787 > [11/May/2023:11:17:07.601] http-in backend_http/mattermost - 0/0/1/1/4 200 > 99492 - - ---- 12/1/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:27:20 gateway haproxy[249887]: 188.29.25.76:5444 > [11/May/2023:11:27:20.445] http-in backend_http/mattermost - 0/0/15/7/35 > 200 99492 - - ---- 22/1/0/0/0 0/0 "GET > /static/5280.1241ee858d210c2d16c9.css HTTP/1.1" > May 11 11:27:51 gateway haproxy[249887]: 188.29.25.76:5446 > [11/May/2023:11:27:51.209] http-in backend_http/mattermost - 0/0/1/1/3 200 > 99492 - - ---- 22/1/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > May 11 11:28:04 gateway haproxy[249887]: 188.29.25.76:1767 > [11/May/2023:11:28:04.311] http-in backend_http/mattermost - 0/0/1/1/3 200 > 99492 - - ---- 21/1/0/0/0 0/0 "GET /static/5280.1241ee858d210c2d16c9.css > HTTP/1.1" > > As for traces, since we can replicate the issue consistently, I'm happy to > do whatever is needed to help diagnose the issue further. Do you happen to > have specific instructions on what you would need? > > Thanks. > > > On Thu, May 11, 2023 at 8:17 PM Willy Tarreau <w...@1wt.eu> wrote: > >> On Thu, May 11, 2023 at 12:34:18PM -0700, Mike Benoit wrote: >> > A specific web application that uses large 99.5KB .CSS files is causing >> a >> > net::ERR_INCOMPLETE_CHUNKED_ENCODING when being accessed from a >> computer on >> > a high latency network (across the Atlantic ocean). We are not able to >> > replicate the problem from any closer devices, but multiple computers >> (and >> > phones) using multiple separate internet connections across the Atlantic >> > exhibit the same issue. >> > >> > We ran into similar issues with H2 and H1 over SSL, though it was >> > net::ERR_CONNECTION_RESET errors instead. We eventually replicated the >> > issue on H1 with SSL off to minimize the variables. >> > >> > If we bypass HAProxy and use TCP port forwarding directly to the backend >> > web server, the issue no longer occurs. So it seems to be related to >> > HAProxy being in the mix, though I don't see any errors in the logs. >> > >> > Packet captures from both ends and config files are available for >> download >> > here: >> > >> https://drive.google.com/file/d/1nrHdsdpXyMi5uAGAQ7HZBSZd-x0pYVQP/view?usp=sharing >> > >> > We are using HAProxy 2.7.7 on Ubuntu 22.04. >> > >> > Any ideas what may be causing this issue? >> >> Here, 1056 bytes are missing from the output in both cases. What do you >> have in haproxy's logs for these requests ? It might be possible that >> the server returns an invalid final chunk for example, that haproxy fails >> to parse, resulting in truncated output. Another possibility might be >> that the server connection experienced an error just after the end of >> the transfer and that for some reason haproxy failed to retrieve the >> last bytes of the stream. I tried to fetch your CSS file from here >> using openssl s_client (to avoid any interpretation from a browser) and >> I'm properly seeing the 0 CRLF CRLF at the end so it's still not clear >> what might be causing this. >> >> Depending on your traffic and the frequency of this issue, maybe it >> could be worth enabling traces at developer level: they'll reveal each >> and every point in the H1/H2 muxes where we passed and might help figure >> why the last chunk is missing. But as a first step, looking at the logs >> to see if haproxy considered it closed normally, aborted on client or >> on server will be of significant help. >> >> Willy >> >
