Hi Maciej,
On Tue, May 07, 2019 at 07:08:47PM +0200, Maciej Zdeb wrote:
> Hi,
>
> I've got another bug with 100% CPU on HAProxy process, it is built from
> HEAD of 1.9 branch.
>
> One of processes stuck in infinite loop, admin socket is not responsive so
> I've got information only from gdb:
>
> 0x0000000000484ab8 in h2_process_mux (h2c=0x2e8ff30) at src/mux_h2.c:2589
> 2589 if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
> (gdb) n
(...)
CCing Olivier. Olivier, I'm wondering if this is not directly related to
what you addressed with this fix merged in 2.0 but not backported :
998410a ("BUG/MEDIUM: h2: Revamp the way send subscriptions works.")
>From what I'm seeing there's no error, the stream is in the sending list,
there's no blocking flag, well, everything looks OK, but we're looping
on SUB_CALL_UNSUBSCRIBE which apprently should not if I understand it
right. Do you think we should backport this patch ?
Remaining of the trace below for reference.
THanks,
Willy
---
> 2587 if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb)
> 2586 list_for_each_entry(h2s, &h2c->send_list, list) {
> (gdb)
> 2587 if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb)
> 2589 if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
> (gdb)
> 2587 if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb)
> 2586 list_for_each_entry(h2s, &h2c->send_list, list) {
> (gdb)
> 2587 if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb) p h2c
> $1 = (struct h2c *) 0x2e8ff30
> (gdb) p *h2c
> $2 = {conn = 0x2b4c900, st0 = H2_CS_FRAME_H, errcode = H2_ERR_NO_ERROR,
> flags = 0, streams_limit = 100, max_id = 149, rcvd_c = 0, rcvd_s = 0, ddht
> = 0x34099c0, dbuf = {size = 0, area = 0x0, data = 0, head = 0}, dsi = 149,
> dfl = 0,
> dft = 1 '\001', dff = 37 '%', dpl = 0 '\000', last_sid = -1, mbuf = {size
> = 16384, area = 0x2ec3d50 "", data = 0, head = 0}, msi = -1, mfl = 0, mft =
> 0 '\000', mff = 0 '\000', miw = 6291456, mws = 15443076, mfs = 16384,
> timeout = 20000, shut_timeout = 20000, nb_streams = 53, nb_cs = 53,
> nb_reserved = 0, stream_cnt = 75, proxy = 0x219ffe0, task = 0x34081d0,
> streams_by_id = {b = {0x2adc2e1, 0x0}}, send_list = {n = 0x2ac5b38, p =
> 0x3093c18},
> fctl_list = {n = 0x2e90008, p = 0x2e90008}, sending_list = {n =
> 0x2ac5b48, p = 0x2ec2798}, buf_wait = {target = 0x0, wakeup_cb = 0x0, list
> = {n = 0x2e90038, p = 0x2e90038}}, wait_event = {task = 0x2b2ae90, handle =
> 0x0, events = 1}}
> (gdb) n
> 2589 if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
> (gdb) p *h2s
> $3 = {cs = 0x297bdb0, sess = 0x819580 <pool_cache+96>, h2c = 0x2e8ff30, h1m
> = {state = H1_MSG_RPBEFORE, flags = 12, curr_len = 0, body_len = 0, next =
> 0, err_pos = -1, err_state = 0}, by_id = {node = {branches = {b =
> {0x2a72250,
> 0x2961c30}}, node_p = 0x2a72251, leaf_p = 0x2961c31, bit = 1, pfx
> = 49017}, key = 103}, id = 103, flags = 16385, mws = 6291456, errcode =
> H2_ERR_NO_ERROR, st = H2_SS_HREM, status = 0, body_len = 0, rxbuf = {size =
> 0,
> area = 0x0, data = 0, head = 0}, wait_event = {task = 0x2fb3ee0, handle
> = 0x0, events = 0}, recv_wait = 0x2b8d700, send_wait = 0x2b8d700, list = {n
> = 0x3130108, p = 0x2b02238}, sending_list = {n = 0x3130118, p = 0x2b02248}}
> (gdb) n
> 2587 if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb) p *h2c
> $4 = {conn = 0x2b4c900, st0 = H2_CS_FRAME_H, errcode = H2_ERR_NO_ERROR,
> flags = 0, streams_limit = 100, max_id = 149, rcvd_c = 0, rcvd_s = 0, ddht
> = 0x34099c0, dbuf = {size = 0, area = 0x0, data = 0, head = 0}, dsi = 149,
> dfl = 0,
> dft = 1 '\001', dff = 37 '%', dpl = 0 '\000', last_sid = -1, mbuf = {size
> = 16384, area = 0x2ec3d50 "", data = 0, head = 0}, msi = -1, mfl = 0, mft =
> 0 '\000', mff = 0 '\000', miw = 6291456, mws = 15443076, mfs = 16384,
> timeout = 20000, shut_timeout = 20000, nb_streams = 53, nb_cs = 53,
> nb_reserved = 0, stream_cnt = 75, proxy = 0x219ffe0, task = 0x34081d0,
> streams_by_id = {b = {0x2adc2e1, 0x0}}, send_list = {n = 0x2ac5b38, p =
> 0x3093c18},
> fctl_list = {n = 0x2e90008, p = 0x2e90008}, sending_list = {n =
> 0x2ac5b48, p = 0x2ec2798}, buf_wait = {target = 0x0, wakeup_cb = 0x0, list
> = {n = 0x2e90038, p = 0x2e90038}}, wait_event = {task = 0x2b2ae90, handle =
> 0x0, events = 1}}
> (gdb) n
> 2586 list_for_each_entry(h2s, &h2c->send_list, list) {
> (gdb) n
> 2587 if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb) n
> 2589 if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
>
>
> HAProxy info:
> HA-Proxy version 1.9.7-207ba5a 2019/05/05 - https://haproxy.org/
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
> -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
> -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
> -Wno-missing-field-initializers -Wtype-limits -DIP_BIND_ADDRESS_NO_PORT=24
> OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_DL=1
> USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1
>
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Built with OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
> Running on OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> Built with Lua version : Lua 5.3.5
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Built with zlib version : 1.2.8
> Running on zlib version : 1.2.8
> Compression algorithms supported : identity("identity"),
> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with PCRE version : 8.43 2019-02-23
> Running on PCRE version : 8.43 2019-02-23
> PCRE library supports JIT : yes
> Encrypted password support via crypt(3): yes
> Built with multi-threading support.
>
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
>
> Available multiplexer protocols :
> (protocols marked as <default> cannot be specified using 'proto' keyword)
> h2 : mode=HTX side=FE|BE
> h2 : mode=HTTP side=FE
> <default> : mode=HTX side=FE|BE
> <default> : mode=TCP|HTTP side=FE|BE
>
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [CACHE] cache
> [TRACE] trace
