On Thu, May 11, 2023 at 12:34:18PM -0700, Mike Benoit wrote: > A specific web application that uses large 99.5KB .CSS files is causing a > net::ERR_INCOMPLETE_CHUNKED_ENCODING when being accessed from a computer on > a high latency network (across the Atlantic ocean). We are not able to > replicate the problem from any closer devices, but multiple computers (and > phones) using multiple separate internet connections across the Atlantic > exhibit the same issue. > > We ran into similar issues with H2 and H1 over SSL, though it was > net::ERR_CONNECTION_RESET errors instead. We eventually replicated the > issue on H1 with SSL off to minimize the variables. > > If we bypass HAProxy and use TCP port forwarding directly to the backend > web server, the issue no longer occurs. So it seems to be related to > HAProxy being in the mix, though I don't see any errors in the logs. > > Packet captures from both ends and config files are available for download > here: > https://drive.google.com/file/d/1nrHdsdpXyMi5uAGAQ7HZBSZd-x0pYVQP/view?usp=sharing > > We are using HAProxy 2.7.7 on Ubuntu 22.04. > > Any ideas what may be causing this issue?
Here, 1056 bytes are missing from the output in both cases. What do you have in haproxy's logs for these requests ? It might be possible that the server returns an invalid final chunk for example, that haproxy fails to parse, resulting in truncated output. Another possibility might be that the server connection experienced an error just after the end of the transfer and that for some reason haproxy failed to retrieve the last bytes of the stream. I tried to fetch your CSS file from here using openssl s_client (to avoid any interpretation from a browser) and I'm properly seeing the 0 CRLF CRLF at the end so it's still not clear what might be causing this. Depending on your traffic and the frequency of this issue, maybe it could be worth enabling traces at developer level: they'll reveal each and every point in the H1/H2 muxes where we passed and might help figure why the last chunk is missing. But as a first step, looking at the logs to see if haproxy considered it closed normally, aborted on client or on server will be of significant help. Willy
