Forgot to attach information about HAProxy (for tracing the issue I compiled it with debug symbols and without optimizations):
haproxy -vv HA-Proxy version 1.9.7 2019/04/25 - https://haproxy.org/ Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -DIP_BIND_ADDRESS_NO_PORT=24 OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_DL=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019 Running on OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.5 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with PCRE version : 8.43 2019-02-23 Running on PCRE version : 8.43 2019-02-23 PCRE library supports JIT : yes Encrypted password support via crypt(3): yes Built with multi-threading support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE h2 : mode=HTX side=FE|BE <default> : mode=HTX side=FE|BE <default> : mode=TCP|HTTP side=FE|BE Available filters : [SPOE] spoe [COMP] compression [CACHE] cache [TRACE] trace wt., 30 kwi 2019 o 08:16 Maciej Zdeb <mac...@zdeb.pl> napisał(a): > Hi, > > I'm returning with problem similar as in thread "[1.9.6] One of haproxy > processes using 100% CPU". > > Thanks to Olivier hard work, some issues where fixed but still not all of > them. :( Currently it is much harder to trigger and it occurs on https > virtual server and not tcp one. I'm observing this problem for HAProxy > 1.9.7, process starts using 100% cpu and admin socket is still responsive. > > Attached gdb session. Please let me know if you need some more info from > gdb, unfortunately I'm not gdb expert and I'm not sure what to look for. It > is production server, however I can keep it for a couple hours isolated > from production traffic. > > Pasting some anonymized info from show sess, please note that it was > dumped on 30/Apr/2019:06:53:54 (before dump, traffic was switched to > another server) but some connections are from 29/Apr/2019 and persist in > that strange state. > > CC: Oliver :) > > socat /var/run/haproxy/haproxy1.sock - <<< "show sess all" > 0x56bb4f0: [29/Apr/2019:12:31:37.186578] id=14500574 proto=tcpv4 > source=C.C.C.C:56567 > flags=0x44e, conn_retries=2, srv_conn=0x24db060, pend_pos=(nil) waiting=0 > frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) > addr=V.V.V.V:443 > backend=V.V.V.V:443_back (id=46 mode=http) addr=S.S.S.S:57654 > server=slot_9_3 (id=78) addr=B.B.B.B:31160 > task=0x6ec4070 (state=0x00 nice=0 calls=2 exp=<NEVER> tmask=0x1 > age=18h22m) > txn=0x57eadf0 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st > =MSG_RPBEFORE > req.f=0x0c blen=0 chnk=0 next=0 > rsp.f=0x00 blen=0 chnk=0 next=0 > si[0]=0x56bb788 (state=EST flags=0x48008 endp0=CS:0x6f373d0 exp=<NEVER> > et=0x000 sub=0) > si[1]=0x56bb7c8 (state=CON flags=0x111 endp1=CS:0x53a9250 exp=<PAST> > et=0x008 sub=2) > co0=0x533d6e0 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM > target=LISTENER:0x23eb040 > flags=0x80243300 fd=660 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x6f373d0 csf=0x00106a00 ctx=0x70f1e80 > co1=0x91263b0 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM > target=SERVER:0x24db060 > flags=0x00403370 fd=51 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x53a9250 csf=0x00000200 ctx=(nil) > req=0x56bb500 (f=0x4cc00000 an=0x8000 pipe=0 tofwd=0 total=1610) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x56bb508 data=0x58ba660 o=1831 p=1831 req.next=0 i=0 size=16384 > res=0x56bb560 (f=0x80000000 an=0x0 pipe=0 tofwd=0 total=0) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x56bb568 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 > 0x639a150: [29/Apr/2019:12:31:53.231220] id=14501038 proto=tcpv4 > source=C2.C2.C2.C2:30107 > flags=0x44e, conn_retries=2, srv_conn=0x24cdad0, pend_pos=(nil) waiting=0 > frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) > addr=V.V.V.V:443 > backend=V.V.V.V:443_back (id=46 mode=http) addr=S2.S2.S2.S2:8760 > server=slot_9_2 (id=61) addr=B.B.B.B:31160 > task=0x6625ab0 (state=0x00 nice=0 calls=2 exp=<NEVER> tmask=0x1 > age=18h22m) > txn=0x5af9a10 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st > =MSG_RPBEFORE > req.f=0x0c blen=0 chnk=0 next=0 > rsp.f=0x00 blen=0 chnk=0 next=0 > si[0]=0x639a3e8 (state=EST flags=0x48008 endp0=CS:0x64d33f0 exp=<NEVER> > et=0x000 sub=0) > si[1]=0x639a428 (state=CON flags=0x111 endp1=CS:0x5d681c0 exp=<PAST> > et=0x008 sub=2) > co0=0x6f23720 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM > target=LISTENER:0x23eb040 > flags=0x80243300 fd=1282 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x64d33f0 csf=0x00106a00 ctx=0x5cc0890 > co1=0x6039c60 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM > target=SERVER:0x24cdad0 > flags=0x00403370 fd=577 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x5d681c0 csf=0x00000200 ctx=0x5d681e0 > req=0x639a160 (f=0x4cc00000 an=0x8000 pipe=0 tofwd=0 total=1137) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x639a168 data=0x5408ef0 o=1349 p=1349 req.next=0 i=0 size=16384 > res=0x639a1c0 (f=0x80000000 an=0x0 pipe=0 tofwd=0 total=0) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x639a1c8 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 > 0x882c0f0: [29/Apr/2019:12:31:46.503263] id=14503967 proto=tcpv4 source= > 5.173.79.240:7434 > flags=0x44e, conn_retries=2, srv_conn=0x24cdad0, pend_pos=(nil) waiting=0 > frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) > addr=V.V.V.V:443 > backend=V.V.V.V:443_back (id=46 mode=http) addr=S2.S2.S2.S2:24688 > server=slot_9_2 (id=61) addr=B.B.B.B:31160 > task=0x7c13f50 (state=0x00 nice=0 calls=2 exp=<NEVER> tmask=0x1 > age=18h22m) > txn=0x5978210 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st > =MSG_RPBEFORE > req.f=0x0c blen=0 chnk=0 next=0 > rsp.f=0x00 blen=0 chnk=0 next=0 > si[0]=0x882c388 (state=EST flags=0x48008 endp0=CS:0x7901a20 exp=<NEVER> > et=0x000 sub=0) > si[1]=0x882c3c8 (state=CON flags=0x111 endp1=CS:0x6ca3a50 exp=<PAST> > et=0x008 sub=2) > co0=0x67684b0 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM > target=LISTENER:0x23eb040 > flags=0x80243300 fd=970 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x7901a20 csf=0x00106a00 ctx=0x7ac8650 > co1=0x74cba90 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM > target=SERVER:0x24cdad0 > flags=0x00403370 fd=2185 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x6ca3a50 csf=0x00000200 ctx=(nil) > req=0x882c100 (f=0x4cc00000 an=0x8000 pipe=0 tofwd=0 total=2096) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x882c108 data=0x5f42c30 o=2307 p=2307 req.next=0 i=0 size=16384 > res=0x882c160 (f=0x80000000 an=0x0 pipe=0 tofwd=0 total=0) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x882c168 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 > 0x77af230: [29/Apr/2019:16:05:25.433552] id=26109777 proto=tcpv4 > source=C3.C3.C3.C3:19974 > flags=0x4e, conn_retries=2, srv_conn=0x24cb520, pend_pos=(nil) waiting=0 > frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) > addr=V.V.V.V:443 > backend=V.V.V.V:443_back (id=46 mode=http) addr=S2.S2.S2.S2:55296 > server=slot_6_2 (id=58) addr=B2.B2.B2.B2:31165 > task=0x54bce70 (state=0x00 nice=0 calls=2 exp=<NEVER> tmask=0x1 > age=14h48m) > txn=0x5f57d30 flags=0x88000000 meth=3 status=-1 req.st=MSG_DONE rsp.st > =MSG_RPBEFORE > req.f=0x0d blen=66 chnk=0 next=0 > rsp.f=0x00 blen=0 chnk=0 next=0 > si[0]=0x77af4c8 (state=EST flags=0x48008 endp0=CS:0x5943120 exp=<NEVER> > et=0x000 sub=0) > si[1]=0x77af508 (state=CON flags=0x111 endp1=CS:0x7a69aa0 exp=<PAST> > et=0x008 sub=2) > co0=0x6694b60 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM > target=LISTENER:0x23eb040 > flags=0x80243300 fd=699 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x5943120 csf=0x00106a00 ctx=0x5931bf0 > co1=0x6af1500 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM > target=SERVER:0x24cb520 > flags=0x00403370 fd=1574 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x7a69aa0 csf=0x00000200 ctx=(nil) > req=0x77af240 (f=0x4cc00000 an=0x8000 pipe=0 tofwd=0 total=1183) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x77af248 data=0x540cf00 o=1398 p=1398 req.next=0 i=0 size=16384 > res=0x77af2a0 (f=0x80000000 an=0x0 pipe=0 tofwd=0 total=0) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x77af2a8 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 > 0x5d1cf90: [29/Apr/2019:18:54:15.974874] id=35265590 proto=tcpv4 > source=C4.C4.C4.C4:55010 > flags=0x44e, conn_retries=2, srv_conn=0x24bb9e0, pend_pos=(nil) waiting=0 > frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) > addr=V.V.V.V:443 > backend=V.V.V.V:443_back (id=46 mode=http) addr=10.32.189.89:17092 > server=slot_3_1 (id=38) addr=B3.B3.B3.B3:31565 > task=0x9848fb0 (state=0x00 nice=0 calls=2 exp=<NEVER> tmask=0x1 > age=11h59m) > txn=0x5540d70 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st > =MSG_RPBEFORE > req.f=0x0c blen=0 chnk=0 next=0 > rsp.f=0x00 blen=0 chnk=0 next=0 > si[0]=0x5d1d228 (state=EST flags=0x48008 endp0=CS:0x905b1b0 exp=<NEVER> > et=0x000 sub=0) > si[1]=0x5d1d268 (state=CON flags=0x111 endp1=CS:0x5474790 exp=<PAST> > et=0x008 sub=2) > co0=0x53a5690 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM > target=LISTENER:0x23eb040 > flags=0x80243300 fd=10 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x905b1b0 csf=0x00106a00 ctx=0x5907040 > co1=0x67a6ed0 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM > target=SERVER:0x24bb9e0 > flags=0x00403370 fd=1356 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x5474790 csf=0x00000200 ctx=(nil) > req=0x5d1cfa0 (f=0x4cc00000 an=0x8000 pipe=0 tofwd=0 total=1428) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x5d1cfa8 data=0x5bd1c00 o=1646 p=1646 req.next=0 i=0 size=16384 > res=0x5d1d000 (f=0x80000000 an=0x0 pipe=0 tofwd=0 total=0) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x5d1d008 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 > 0x6e5d100: [29/Apr/2019:21:47:45.668692] id=46668022 proto=tcpv4 > source=C5.C5.C5.C5:38289 > flags=0x44e, conn_retries=2, srv_conn=0x24d4be0, pend_pos=(nil) waiting=0 > frontend=virtual-server_front (id=45 mode=http), listener=? (id=1) > addr=V.V.V.V:443 > backend=V.V.V.V:443_back (id=46 mode=http) addr=S.S.S.S:22918 > server=slot_1_3 (id=70) addr=B4.B4.B4.B4:31992 > task=0x59e1a10 (state=0x00 nice=0 calls=2 exp=<NEVER> tmask=0x1 age=9h6m) > txn=0x92b23d0 flags=0x88003000 meth=1 status=-1 req.st=MSG_DONE rsp.st > =MSG_RPBEFORE > req.f=0x0c blen=0 chnk=0 next=0 > rsp.f=0x00 blen=0 chnk=0 next=0 > si[0]=0x6e5d398 (state=EST flags=0x48008 endp0=CS:0x7e027a0 exp=<NEVER> > et=0x000 sub=0) > si[1]=0x6e5d3d8 (state=CON flags=0x111 endp1=CS:0x6459520 exp=<PAST> > et=0x008 sub=2) > co0=0x6de8980 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM > target=LISTENER:0x23eb040 > flags=0x80243300 fd=3041 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x7e027a0 csf=0x00106a00 ctx=0x82f5900 > co1=0xb84e280 ctrl=tcpv4 xprt=RAW mux=PASS data=STRM > target=SERVER:0x24d4be0 > flags=0x00403370 fd=1787 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x6459520 csf=0x00000200 ctx=0x6459540 > req=0x6e5d110 (f=0x4cc00000 an=0x8000 pipe=0 tofwd=0 total=1112) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x6e5d118 data=0x7235570 o=1324 p=1324 req.next=0 i=0 size=16384 > res=0x6e5d170 (f=0x80000000 an=0x0 pipe=0 tofwd=0 total=0) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x6e5d178 data=(nil) o=0 p=0 rsp.next=0 i=0 size=0 > 0x680b1b0: [30/Apr/2019:06:53:54.039969] id=62359636 proto=unix_stream > source=unix:1 > flags=0x8, conn_retries=0, srv_conn=(nil), pend_pos=(nil) waiting=0 > frontend=GLOBAL (id=0 mode=tcp), listener=? (id=1) addr=unix:1 > backend=<NONE> (id=-1 mode=-) > server=<NONE> (id=-1) > task=0x67a4f20 (state=0x00 nice=-64 calls=1 exp=<NEVER> tmask=0x1 age=0s) > si[0]=0x680b448 (state=EST flags=0x80008 endp0=CS:0x660fdf0 exp=<NEVER> > et=0x000 sub=0) > si[1]=0x680b488 (state=EST flags=0x204018 endp1=APPCTX:0x53a1a30 > exp=<NEVER> et=0x000 sub=0) > co0=0x8a5b8f0 ctrl=unix_stream xprt=RAW mux=PASS data=STRM > target=LISTENER:0x19988b0 > flags=0x00243300 fd=17 fd.state=22 fd.cache=0 updt=0 fd.tmask=0x1 > cs=0x660fdf0 csf=0x00001000 ctx=0x660fe10 > app1=0x53a1a30 st0=7 st1=0 st2=3 applet=<CLI> tmask=0x1, nice=-64, > calls=1, cpu=0, lat=0 > req=0x680b1c0 (f=0xc4c220 an=0x0 pipe=0 tofwd=0 total=14) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x680b1c8 data=0x718bca0 o=0 p=0 req.next=0 i=0 size=16384 > res=0x680b220 (f=0x80008002 an=0x0 pipe=0 tofwd=-1 total=9380) > an_exp=<NEVER> rex=<NEVER> wex=<NEVER> > buf=0x680b228 data=0x9d33d90 o=9380 p=9380 rsp.next=0 i=0 size=16384 > > socat /var/run/haproxy/haproxy1.sock - <<< "show info" > Name: HAProxy > Version: 1.9.7 > Release_date: 2019/04/25 > Nbthread: 1 > Nbproc: 8 > Process_num: 1 > Pid: 44782 > Uptime: 0d 22h54m13s > Uptime_sec: 82453 > Memmax_MB: 0 > PoolAlloc_MB: 9 > PoolUsed_MB: 9 > PoolFailed: 0 > Ulimit-n: 1000000 > Maxsock: 4001570 > Maxconn: 2000000 > Hard_maxconn: 2000000 > CurrConns: 7 > CumConns: 16818215 > CumReq: 62359636 > MaxSslConns: 0 > CurrSslConns: 7 > CumSslConns: 15468203 > Maxpipes: 0 > PipesUsed: 0 > PipesFree: 0 > ConnRate: 0 > ConnRateLimit: 0 > MaxConnRate: 450 > SessRate: 0 > SessRateLimit: 0 > MaxSessRate: 450 > SslRate: 0 > SslRateLimit: 0 > MaxSslRate: 408 > SslFrontendKeyRate: 0 > SslFrontendMaxKeyRate: 367 > SslFrontendSessionReuse_pct: 0 > SslBackendKeyRate: 0 > SslBackendMaxKeyRate: 6 > SslCacheLookups: 5405716 > SslCacheMisses: 2783916 > CompressBpsIn: 0 > CompressBpsOut: 0 > CompressBpsRateLim: 0 > ZlibMemUsage: 0 > MaxZlibMemUsage: 0 > Tasks: 3748 > Run_queue: 3 > Idle_pct: 1 > node: X > Stopping: 0 > Jobs: 898 > Unstoppable Jobs: 0 > Listeners: 890 > ActivePeers: 0 > ConnectedPeers: 0 > DroppedLogs: 134 > BusyPolling: 0 > > socat /var/run/haproxy/haproxy1.sock - <<< "show activity" > thread_id: 0 > date_now: 1556600105.168009 > loops: 730017734 > wake_cache: 220599287 > wake_tasks: 412569304 > wake_signal: 0 > poll_exp: 633168591 > poll_drop: 19816095 > poll_dead: 0 > poll_skip: 0 > fd_skip: 0 > fd_lock: 0 > fd_del: 0 > conn_dead: 0 > stream: 144545285 > empty_rq: 68411039 > long_rq: 454381935 > cpust_ms_tot: 116 > cpust_ms_1s: 0 > cpust_ms_15s: 0 > avg_loop_us: 50 > > strace > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787194709}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787206841}) = 0 > close(6) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787281934}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787295206}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787357244}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787369184}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787430478}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787442467}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787524492}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787537606}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787603519}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787615975}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787680909}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787693527}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787757296}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787770110}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787833630}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787845869}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787909129}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787921129}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787983756}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 787995880}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788058759}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788070708}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788133753}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788145936}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788209051}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788221068}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788284081}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788296179}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788359128}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788371166}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788434304}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788446107}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788509745}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788521850}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788585197}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788597339}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788661289}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788673507}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788736072}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788748221}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788811472}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788823528}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788886604}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788898706}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788960106}) = 0 > epoll_wait(3, {}, 200, 0) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 788972268}) = 0 > clock_gettime(CLOCK_THREAD_CPUTIME_ID, {53716, 789045142}) = 0 > > >
