Hello This is my first post to this mailing list. I am not good at using English, so feel free to ask me if my text is hard to understand.
I noticed haproxy-1.8 sometimes sent incomplete htmls to clients when running haproxy with the attached haproxy.cfg that uses multithread function. # I also attached content01.html and check.html that # are deployed on backend servers. # content01.html is used in the confirmnation test # described below, check.html is for health check # purpose. In this case, the client receives a shorter html. I confirmed haoproxy's log message corresponded to the shorter html, the following line is one of them. 2021-03-23T15:54:46.869626+09:00 lb01 [err] haproxy[703957]: 192.168.1.73:60572 [23/Mar/2021:15:54:46.868] vs_http01 rs_http01web/web01 0/0/0/0/0 200 2896 - - SD-- 12/12/7/2/0 0/0 "GET /content01.html?x=170026897 HTTP/1.1" I ran the curl commnad like the foolowing line at the client to get this log message. $ curl -v "http://192.168.1.21/content01.html?x=[170000000-170499999]"; 1>170000000.sto 2>170000000.ste # The x parameter is for identification. The following line is the log message when a HTTP access is normally done. 2021-03-23T15:54:05.918522+09:00 lb01 [info] haproxy[703957]: 192.168.1.73:44490 [23/Mar/2021:15:54:05.917] vs_http01 rs_http01web/web02 0/0/0/0/0 200 4864 - - ---- 9/9/5/4/0 0/0 "GET /content01.html?x=170000000 HTTP/1.1" The following list is the frequency of abnormal access when a total of 10 million accesses are made in 20 parallel curl processes. status_code bytes_read occurence 200 4344 1 200 2896 9 200 1448 6 408 216 2 The following line is the log messages in the case of 408. 2021-03-23T16:02:42.444084+09:00 lb01 [err] haproxy[703957]: 192.168.1.73:37052 [23/Mar/2021:16:02:32.472] vs_http01 vs_http01/<NOSRV> -1/-1/-1/-1/10000 408 212 - - cR-- 14/14/0/0/0 0/0 "<BADREQ>" When I first met this behavior, I used haproxy-1.8.25 shipped with RHEL8.3. So I obtained haproxy-1.8.30 from http://git.haproxy.org/ and built it, ran the test and got the result of the same sort. This behavior didn't happen without using multithread function. Next, I tried on haproxy-2.0.0 and confirmed it ran normally without this behavior. Then I picked up several versions of haproxy between 1.8.0 and 2.0.0 and built them, test them and found the commit below resolved this behavior. =================================================== commit 524344b4e0434b86d83869ef051f98d24505c08f Author: Olivier Houchard <[email protected]> Date: Wed Sep 12 17:12:53 2018 +0200 MEDIUM: connections: Don't reset the polling flags in conn_fd_handler(). Resetting the polling flags at the end of conn_fd_handler() shouldn't be needed anymore, and it will create problem when we won't handle send/recv from conn_fd_handler() anymore. diff --git a/src/connection.c b/src/connection.c index ab32567b..e303f2c3 100644 --- a/src/connection.c +++ b/src/connection.c @@ -203,9 +203,6 @@ void conn_fd_handler(int fd) conn->mux->wake(conn) < 0) return; - /* remove the events before leaving */ - fdtab[fd].ev &= FD_POLL_STICKY; - /* commit polling changes */ conn->flags &= ~CO_FL_WILL_UPDATE; conn_cond_update_polling(conn); =================================================== So I commented out "fdtab[fd].ev &= FD_POLL_STICKY;" from both of haproxy-1.8.25 and haproxy-1.8.30, then the behavior is resolved. I don't know why this commit resolves the behavior, I just tried and ran the test. My question is: Does this commit make side effects such like data structure corruption or memory leak on haprox-1.8 ? And if someone tell me how the behavior happens and how the commmit resolves it, I will be appreciated it. Best regards. P.S. I also tried haproxy-2.0.22 and I didn't find messages about shorter htmls but the number of log messages is smaller than the number of HTTP accesses. I am considering this would be another issue. Both of haproxy-2.3.0 and haproxy-2.3.10 don't make shorter html issue and insufficient log messages issue. -- Takenaka Kazuhiro email: <[email protected]> or <[email protected]>Title: content01 0001 123456789 123456789 123456789 123456789 123456789X 0002 123456789 123456789 123456789 123456789 123456789X 0003 123456789 123456789 123456789 123456789 123456789X 0004 123456789 123456789 123456789 123456789 123456789X 0005 123456789 123456789 123456789 123456789 123456789X 0006 123456789 123456789 123456789 123456789 123456789X 0007 123456789 123456789 123456789 123456789 123456789X 0008 123456789 123456789 123456789 123456789 123456789X 0009 123456789 123456789 123456789 123456789 123456789X 0010 123456789 123456789 123456789 123456789 123456789X 0011 123456789 123456789 123456789 123456789 123456789X 0012 123456789 123456789 123456789 123456789 123456789X 0013 123456789 123456789 123456789 123456789 123456789X 0014 123456789 123456789 123456789 123456789 123456789X 0015 123456789 123456789 123456789 123456789 123456789X 0016 123456789 123456789 123456789 123456789 123456789X 0017 123456789 123456789 123456789 123456789 123456789X 0018 123456789 123456789 123456789 123456789 123456789X 0019 123456789 123456789 123456789 123456789 123456789X 0020 123456789 123456789 123456789 123456789 123456789X 0021 123456789 123456789 123456789 123456789 123456789X 0022 123456789 123456789 123456789 123456789 123456789X 0023 123456789 123456789 123456789 123456789 123456789X 0024 123456789 123456789 123456789 123456789 123456789X 0025 123456789 123456789 123456789 123456789 123456789X 0026 123456789 123456789 123456789 123456789 123456789X 0027 123456789 123456789 123456789 123456789 123456789X 0028 123456789 123456789 123456789 123456789 123456789X 0029 123456789 123456789 123456789 123456789 123456789X 0030 123456789 123456789 123456789 123456789 123456789X 0031 123456789 123456789 123456789 123456789 123456789X 0032 123456789 123456789 123456789 123456789 123456789X 0033 123456789 123456789 123456789 123456789 123456789X 0034 123456789 123456789 123456789 123456789 123456789X 0035 123456789 123456789 123456789 123456789 123456789X 0036 123456789 123456789 123456789 123456789 123456789X 0037 123456789 123456789 123456789 123456789 123456789X 0038 123456789 123456789 123456789 123456789 123456789X 0039 123456789 123456789 123456789 123456789 123456789X 0040 123456789 123456789 123456789 123456789 123456789X 0041 123456789 123456789 123456789 123456789 123456789X 0042 123456789 123456789 123456789 123456789 123456789X 0043 123456789 123456789 123456789 123456789 123456789X 0044 123456789 123456789 123456789 123456789 123456789X 0045 123456789 123456789 123456789 123456789 123456789X 0046 123456789 123456789 123456789 123456789 123456789X 0047 123456789 123456789 123456789 123456789 123456789X 0048 123456789 123456789 123456789 123456789 123456789X 0049 123456789 123456789 123456789 123456789 123456789X 0050 123456789 123456789 123456789 123456789 123456789X 0051 123456789 123456789 123456789 123456789 123456789X 0052 123456789 123456789 123456789 123456789 123456789X 0053 123456789 123456789 123456789 123456789 123456789X 0054 123456789 123456789 123456789 123456789 123456789X 0055 123456789 123456789 123456789 123456789 123456789X 0056 123456789 123456789 123456789 123456789 123456789X 0057 123456789 123456789 123456789 123456789 123456789X 0058 123456789 123456789 123456789 123456789 123456789X 0059 123456789 123456789 123456789 123456789 123456789X 0060 123456789 123456789 123456789 123456789 123456789X 0061 123456789 123456789 123456789 123456789 123456789X 0062 123456789 123456789 123456789 123456789 123456789X 0063 123456789 123456789 123456789 123456789 123456789X 0064 123456789 123456789 123456789 123456789 123456789X 0065 123456789 123456789 123456789 123456789 123456789X 0066 123456789 123456789 123456789 123456789 123456789X 0067 123456789 123456789 123456789 123456789 123456789X 0068 123456789 123456789 123456789 123456789 123456789X 0069 123456789 123456789 123456789 123456789 123456789X 0070 123456789 123456789 123456789 123456789 123456789X 0071 123456789 123456789 123456789 123456789 123456789X 0072 123456789 123456789 123456789 123456789 123456789X 0073 123456789 123456789 123456789 123456789 123456789X 0074 123456789 123456789 123456789 123456789 123456789X 0075 123456789 123456789 123456789 123456789 123456789X 0076 123456789 123456789 123456789 123456789 123456789X 0077 123456789 123456789 123456789 123456789 123456789X 0078 123456789 123456789 123456789 123456789 123456789X 0079 123456789 123456789 123456789 123456789 123456789X 0080 123456789 123456789 123456789 123456789 123456789X
haproxy.cfg
Description: Binary data
