Hi Maciej,

On Tue, May 07, 2019 at 07:08:47PM +0200, Maciej Zdeb wrote:
> Hi,
> 
> I've got another bug with 100% CPU on HAProxy process, it is built from
> HEAD of 1.9 branch.
> 
> One of processes stuck in infinite loop, admin socket is not responsive so
> I've got information only from gdb:
> 
> 0x0000000000484ab8 in h2_process_mux (h2c=0x2e8ff30) at src/mux_h2.c:2589
> 2589            if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
> (gdb) n
(...)

CCing Olivier. Olivier, I'm wondering if this is not directly related to
what you addressed with this fix merged in 2.0 but not backported :

  998410a ("BUG/MEDIUM: h2: Revamp the way send subscriptions works.")

>From what I'm seeing there's no error, the stream is in the sending list,
there's no blocking flag, well, everything looks OK, but we're looping
on SUB_CALL_UNSUBSCRIBE which apprently should not if I understand it
right. Do you think we should backport this patch ?

Remaining of the trace below for reference.

THanks,
Willy

---

> 2587            if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb)
> 2586        list_for_each_entry(h2s, &h2c->send_list, list) {
> (gdb)
> 2587            if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb)
> 2589            if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
> (gdb)
> 2587            if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb)
> 2586        list_for_each_entry(h2s, &h2c->send_list, list) {
> (gdb)
> 2587            if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb) p h2c
> $1 = (struct h2c *) 0x2e8ff30
> (gdb) p *h2c
> $2 = {conn = 0x2b4c900, st0 = H2_CS_FRAME_H, errcode = H2_ERR_NO_ERROR,
> flags = 0, streams_limit = 100, max_id = 149, rcvd_c = 0, rcvd_s = 0, ddht
> = 0x34099c0, dbuf = {size = 0, area = 0x0, data = 0, head = 0}, dsi = 149,
> dfl = 0,
>   dft = 1 '\001', dff = 37 '%', dpl = 0 '\000', last_sid = -1, mbuf = {size
> = 16384, area = 0x2ec3d50 "", data = 0, head = 0}, msi = -1, mfl = 0, mft =
> 0 '\000', mff = 0 '\000', miw = 6291456, mws = 15443076, mfs = 16384,
>   timeout = 20000, shut_timeout = 20000, nb_streams = 53, nb_cs = 53,
> nb_reserved = 0, stream_cnt = 75, proxy = 0x219ffe0, task = 0x34081d0,
> streams_by_id = {b = {0x2adc2e1, 0x0}}, send_list = {n = 0x2ac5b38, p =
> 0x3093c18},
>   fctl_list = {n = 0x2e90008, p = 0x2e90008}, sending_list = {n =
> 0x2ac5b48, p = 0x2ec2798}, buf_wait = {target = 0x0, wakeup_cb = 0x0, list
> = {n = 0x2e90038, p = 0x2e90038}}, wait_event = {task = 0x2b2ae90, handle =
> 0x0, events = 1}}
> (gdb) n
> 2589            if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
> (gdb) p *h2s
> $3 = {cs = 0x297bdb0, sess = 0x819580 <pool_cache+96>, h2c = 0x2e8ff30, h1m
> = {state = H1_MSG_RPBEFORE, flags = 12, curr_len = 0, body_len = 0, next =
> 0, err_pos = -1, err_state = 0}, by_id = {node = {branches = {b =
> {0x2a72250,
>           0x2961c30}}, node_p = 0x2a72251, leaf_p = 0x2961c31, bit = 1, pfx
> = 49017}, key = 103}, id = 103, flags = 16385, mws = 6291456, errcode =
> H2_ERR_NO_ERROR, st = H2_SS_HREM, status = 0, body_len = 0, rxbuf = {size =
> 0,
>     area = 0x0, data = 0, head = 0}, wait_event = {task = 0x2fb3ee0, handle
> = 0x0, events = 0}, recv_wait = 0x2b8d700, send_wait = 0x2b8d700, list = {n
> = 0x3130108, p = 0x2b02238}, sending_list = {n = 0x3130118, p = 0x2b02248}}
> (gdb) n
> 2587            if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb) p *h2c
> $4 = {conn = 0x2b4c900, st0 = H2_CS_FRAME_H, errcode = H2_ERR_NO_ERROR,
> flags = 0, streams_limit = 100, max_id = 149, rcvd_c = 0, rcvd_s = 0, ddht
> = 0x34099c0, dbuf = {size = 0, area = 0x0, data = 0, head = 0}, dsi = 149,
> dfl = 0,
>   dft = 1 '\001', dff = 37 '%', dpl = 0 '\000', last_sid = -1, mbuf = {size
> = 16384, area = 0x2ec3d50 "", data = 0, head = 0}, msi = -1, mfl = 0, mft =
> 0 '\000', mff = 0 '\000', miw = 6291456, mws = 15443076, mfs = 16384,
>   timeout = 20000, shut_timeout = 20000, nb_streams = 53, nb_cs = 53,
> nb_reserved = 0, stream_cnt = 75, proxy = 0x219ffe0, task = 0x34081d0,
> streams_by_id = {b = {0x2adc2e1, 0x0}}, send_list = {n = 0x2ac5b38, p =
> 0x3093c18},
>   fctl_list = {n = 0x2e90008, p = 0x2e90008}, sending_list = {n =
> 0x2ac5b48, p = 0x2ec2798}, buf_wait = {target = 0x0, wakeup_cb = 0x0, list
> = {n = 0x2e90038, p = 0x2e90038}}, wait_event = {task = 0x2b2ae90, handle =
> 0x0, events = 1}}
> (gdb) n
> 2586        list_for_each_entry(h2s, &h2c->send_list, list) {
> (gdb) n
> 2587            if (h2c->st0 >= H2_CS_ERROR || h2c->flags &
> H2_CF_MUX_BLOCK_ANY)
> (gdb) n
> 2589            if (h2s->send_wait->events & SUB_CALL_UNSUBSCRIBE)
> 
> 
> HAProxy info:
> HA-Proxy version 1.9.7-207ba5a 2019/05/05 - https://haproxy.org/
> Build options :
>   TARGET  = linux2628
>   CPU     = generic
>   CC      = gcc
>   CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
> -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
> -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
> -Wno-missing-field-initializers -Wtype-limits -DIP_BIND_ADDRESS_NO_PORT=24
>   OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_DL=1
> USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1
> 
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Built with OpenSSL version : OpenSSL 1.1.1b  26 Feb 2019
> Running on OpenSSL version : OpenSSL 1.1.1b  26 Feb 2019
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> Built with Lua version : Lua 5.3.5
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Built with zlib version : 1.2.8
> Running on zlib version : 1.2.8
> Compression algorithms supported : identity("identity"),
> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
> Built with PCRE version : 8.43 2019-02-23
> Running on PCRE version : 8.43 2019-02-23
> PCRE library supports JIT : yes
> Encrypted password support via crypt(3): yes
> Built with multi-threading support.
> 
> Available polling systems :
>       epoll : pref=300,  test result OK
>        poll : pref=200,  test result OK
>      select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.
> 
> Available multiplexer protocols :
> (protocols marked as <default> cannot be specified using 'proto' keyword)
>               h2 : mode=HTX        side=FE|BE
>               h2 : mode=HTTP       side=FE
>        <default> : mode=HTX        side=FE|BE
>        <default> : mode=TCP|HTTP   side=FE|BE
> 
> Available filters :
>     [SPOE] spoe
>     [COMP] compression
>     [CACHE] cache
>     [TRACE] trace

Reply via email to