> Am 01.07.2021 um 14:16 schrieb Yann Ylavic <ylavic....@gmail.com>:
>
> On Thu, Jul 1, 2021 at 10:15 AM Stefan Eissing
> <stefan.eiss...@greenbytes.de> wrote:
>>
>>> Am 30.06.2021 um 18:01 schrieb Eric Covener <cove...@gmail.com>:
>>>
>>> On Wed, Jun 30, 2021 at 11:46 AM Stefan Eissing
>>> <stefan.eiss...@greenbytes.de> wrote:
>>>>
>>>> It looks like we stumbled upon an issue in
>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=65402 which concerns the
>>>> life times of our backend connections.
>>>>
>>>> When a frontend connection causes a backend request and drops, our backend
>>>> connection only notifies the loss when it attempts to pass some data. In
>>>> normal http response processing, this is not an issue since response
>>>> chunks are usually coming in quite frequently. Then the proxied connection
>>>> will fail to pass it to an aborted frontend connection and cleanup will
>>>> occur.
>>>>
>>>> However, with such modern shenanigans such as Server Side Events (SSE),
>>>> the request is supposed to be long running and will produce body chunks
>>>> quite infrequently, like every 30 seconds or so. This leaves our proxy
>>>> workers hanging in recv for quite a while and may lead to worker
>>>> exhaustion.
>>>>
>>>> We can say SSE is a bad idea anyway, but that will probably not stop
>>>> people from doing such crazy things.
>>>>
>>>> What other mitigations do we have?
>>>> - pthread_kill() will interrupt the recv and probably make it fail
>>>> - we can use shorter socket timeouts on backend and check r->connection
>>>> status in between
>>>> - ???
>
> Can mod_proxy_http2 do better here? I suppose we lose all
> relationships in h2->h1 and then h1->h2, asking just in case..
I have not tested, but my guess is that it goes into a blocking read on the
backend as well, since there is nothing it wants to send when a response body
is incoming.
>>>
>>>
>>> In trunk the tunnelling side of mod_proxy_http can go async and get
>>> called back for activity on either side by asking Event to watch both
>>> sockets.
>>
>>
>> How does that work, actually? Do we have an example somewhere?
>
> This is the ap_proxy_tunnel_create() and ap_proxy_tunnel_run()
> called/used by mod_proxy_http for Upgrade(d) protocols.
>
> I'm thinking to improve this interface to have a hook called in
> ap_proxy_transfer_between_connections() with the data being forwarded
> from one side to the other (in/out connection), and the hook could
> decide to let the data pass, or retain them, and/or switch to
> speculative mode, and/or remove/add one side/sense from the pollset,
> or abort, or.. The REALLY_LAST hook would be something like the
> existing ap_proxy_buckets_lifetime_transform().
>
> The hook(s) would be responsible (each) of their connections' states,
> mod_proxy_http could then be implemented fully async in a callback,
> but I suppose mod_h2 could hook itself there too if it has something
> to care about.
Just had a glimpse and it looks interesting, not only for "real" backends but
maybe also for handling h2 workers from a main connection. See below.
>
>>
>>> I'm not sure how browsers treat the SSE connection, can it ever have a
>>> subsequent request? If not, maybe we could see the SSE Content-Type
>>> and shoehorn it into the tunneling (figuring out what to do with
>>> writes from the client, backport the event and async tunnel stuff?)
>>
>> I don't think they will do a subsequent request in the HTTP/1.1 sense,
>> meaning they'll close their H1 connection and open a new one. In H2 land,
>> the request connection is a virtual "secondary" one away.
>
> The issue I see here for inbound h2 is that the tunneling loop needs
> something to poll() on both sides, and there is no socket on the h2
> slave connections to do that.. How to poll a h2 stream, pipe or
> something?
More "something". The plan to switch the current polling to something
pipe based has long stalled. Mainly due to lack of time, but also missing
a bright idea how the server in general should handle such constructs.
>
>>
>> But changing behaviour based on the content type seems inadequate. When
>> the server proxies applications (like uwsgi), the problem may also happen
>> to requests that are slow producing responses.
>>
>> To DoS such a setup, where a proxied response takes n seconds, you'd need
>> total_workers / n aborted requests per second. In HTTP/1.1 that would
>> all be connections and maybe noticeable from a supervisor, but in H2 this
>> could happen all on the same tcp connection (although our h2 implementation
>> has some protection against abusive client behaviour).
>>
>> A general solution to the problem would therefore be valuable, imo.
>
> The general/generic solution for anything proxy could be the tunneling
> loop, a bit like a proxy_tcp (or proxy_transport) module to hook to.
>
>>
>> We should think about solving this in the context of mpm_event, which
>> I believe is the production recommended setup that merits our efforts.
>
> Yes, the tunneling loop stops and the poll()ing is deferred to MPM
> event (the ap_hook_mpm_register_poll_callback*() API) when nothing
> comes from either side for an AsyncDelay.
>
>>
>> If mpm_event could make the link between one connection to another,
>> like frontend to backend, it could wake up backends on a frontend
>> termination. Do you agree, Yann?
>
> Absolutely, but there's more work to be done to get there :)
>
> Also, is this kind of architecture what we really want?
> Ideas, criticisms and discussions welcome!
>
>>
>> Could this be as easy as adding another "conn_rec *context" field
>> in conn_rec that tracks this?
>
> Tracking some connection close (at transport level) on the client side
> to "abort" the transaction is not enough, a connection can be
> half-closed and still want some response back for instance.
>
> We want something that says abort (like h2 RST_STREAM), no such thing
> in transactional HTTP/1 (AFAICT), that's why mod_h2 would need to hook
> in the tunnel if it wanted to abort the loop (IIUC).
Makes sense. There is also the H2 stream state which goes into
HALF_CLOSED_REMOTE when the request is fully sent and a response
is expected to come until the server closes its side of the stream.
The TLS CLOSE_NOTIFY would be an equivalent signal available on a http/1.1
https: connection, I assume.
Cheers, Stefan
>
>
> Cheers;
> Yann.
