mrdg opened a new issue, #9179: URL: https://github.com/apache/trafficserver/issues/9179
We run a proxy server in front of ATS and noticed this problem when the number of streams on a connections gets close to the maximum, and some streams are canceled by the proxy (generally because the original request was canceled). I posted some test code and logs of a reproduction case in [this gist](https://gist.github.com/mrdg/240bb61a0df2b17b8cc9148c4d540ed9). I tested this on master with `proxy.config.http2.max_concurrent_streams_in` set to 5 to make it easier to parse the logs, but we see the issue with much higher limits as well. The test client sends 5 requests, then cancels those, and then sends 5 more, which should be accepted, but it appears that ATS is refusing them. The client code logs this: ``` 2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=11 len=4 ErrCode=REFUSED_STREAM 2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=13 len=4 ErrCode=REFUSED_STREAM 2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=15 len=4 ErrCode=REFUSED_STREAM 2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=17 len=4 ErrCode=REFUSED_STREAM 2022/11/04 12:38:26 http2: Transport received RST_STREAM stream=19 len=4 ErrCode=REFUSED_STREAM ``` Looking at the ATS logs, I see that the previously canceled streams are deleted after the second batch of requests is rejected: ``` [Nov 4 12:38:26.285] [ET_NET 1] DEBUG: <Http2CommonSession.cc:250 (do_start_frame_read)> (http2_cs) [1] receiving frame header [Nov 4 12:38:26.285] [ET_NET 1] DEBUG: <Http2CommonSession.cc:267 (do_start_frame_read)> (http2_cs) [1] frame header length=10, type=1, flags=0x5, streamid=11 [Nov 4 12:38:26.285] [ET_NET 1] DEBUG: <Http2ConnectionState.cc:232 (rcv_headers_frame)> (http2_con) [1] [11] Received HEADERS frame [Nov 4 12:38:26.286] [ET_NET 1] DEBUG: <Http2ConnectionState.cc:1936 (send_rst_stream_frame)> (http2_con) [1] [11] Send RST_STREAM frame ... [Nov 4 12:38:26.288] [ET_NET 1] DEBUG: <Http2Stream.cc:68 (~Http2Stream)> (http2_stream) [1] [1] Destroy stream, sent 0 bytes [Nov 4 12:38:26.288] [ET_NET 1] DEBUG: <Http2ConnectionState.cc:1495 (delete_stream)> (http2_con) [1] [1] Delete stream ``` We also see this in the logs: ``` [Nov 4 12:38:26.287] [ET_NET 1] ERROR: HTTP/2 stream error code=0x07 client_ip=127.0.0.1 session_id=1 stream_id=19 recv headers creating inbound stream beyond max_concurrent limit ``` We also see this compression error (or sometimes a protocol error) when the problem happens: ``` [Nov 4 12:38:26.288] [ET_NET 1] ERROR: HTTP/2 connection error code=0x09 client_ip=127.0.0.1 session_id=1 stream_id=21 recv headers compression error ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
