Re: async mpms + http2

[Stefan Eissing]

> Am 15.12.2015 um 17:58 schrieb Graham Leggett :
> 
> On 15 Dec 2015, at 5:44 PM, Stefan Eissing  
> wrote:
> 
>> However, there is another state in HTTP/2 processing, where a BLOCK_READ is 
>> performed: flow control on streaming out responses. Basically, the client 
>> has a window of n bytes, the server sends n response bytes and then needs to 
>> wait for the client to increase the window again. This would be a good time 
>> to resume processing back to the mpm, and initially I did so. But that 
>> breaks under load.
>> 
>> Under load, the event mpm feels free to close connections in certain states 
>> - I think CONN_STATE_CHECK_REQUEST_LINE_READABLE. That is fine for a HTTP/1 
>> connection as it always is between requests here. But HTTP/2 would also like 
>> to use read events, but unloading a child by closing such connections is not 
>> the right thing to do.
>> 
>> Is there a way mod_http2 can achieve this or do we need another flag in 
>> conn_rec/cs?
> 
> Does the CONN_STATE_WRITE_COMPLETION state fit this case?

As far as I figured out, that will only work when http2 processing moves into 
an output filter. Then the filter would have to mimic pending data and sense == 
READ. That would prevent event from closing the connection early and POLL_IN. 
Next attempt to write the output_filters could then read again.

Other attempts to keep the connection although output filters are empty fail 
under load. I'll give the http2 output filter a try, but it feels a bit like 
hacking the current design (the design is not wrong, it just could not predict 
h2 usage). 

> The WRITE in the above name is a bit of a misnomer, because when mod_ssl 
> returns WANTS_READ we return to the CONN_STATE_WRITE_COMPLETION state with 
> the sense set to READ. This tells the MPM to block waiting until we’re 
> readable, as opposed to block waiting until we’re writable.

I think this only happens, as long as data remains in the output filter.

>> Observation 2: controlled close
>> HTTP/2 would like to send out a last frame when closing a connection in a 
>> controlled/timeout way. Do we need another hook for that? Something like 
>> "pre_close_connection”?
> 
> I’m assuming we would like a module to have control over the sending of that 
> frame, if so, it makes sense yes.

Ok.

>> Observation 3: timeouts
>> a. If admins would like to have another Timeout for HTTP/2 connections, they 
>> are currently stuck with configuring one Timeout which goes to 
>> server_rec->timeout, right? For idle connections, clients already try to 
>> keep it open longer. It would be nice to give admins a choice here for 
>> different values based on the selected protocol. How to best do this?
> 
> Perhaps just as we can override the sense of the connection before returning, 
> we might be able to modify the timeout before returning.

I'd like that.

>> b. Handling own timeouts: if I want, during processing the connection, a 
>> different socket timeout, I can just set it using apr_socket_timeout_set()? 
>> Shall I restore it back to server->timeout before returning?
> 
> That feels wrong, ideally we want to be formally telling the MPM “give us 
> this timeout” rather than doing something sneaky behind it’s back.

Yes. Do not like it either.

>> c. What would be the best way to wait for a file description *and* 
>> conditional event simultaneously? mod_http2 has now a sort of pool loop with 
>> double backup timer for situations where it needs to watch both client and 
>> backend processes.
> 
> I’m not sure how portable it is watching for file descriptors. I have used 
> libev to do this in the past and it was quite involved. However - if we can 
> support this it would be really useful.
> 
> Regards,
> Graham
> —

Cheers,
  Stefan

Re: async mpms + http2

[Graham Leggett]

On 15 Dec 2015, at 5:44 PM, Stefan Eissing  wrote:

>  However, there is another state in HTTP/2 processing, where a BLOCK_READ is 
> performed: flow control on streaming out responses. Basically, the client has 
> a window of n bytes, the server sends n response bytes and then needs to wait 
> for the client to increase the window again. This would be a good time to 
> resume processing back to the mpm, and initially I did so. But that breaks 
> under load.
> 
>  Under load, the event mpm feels free to close connections in certain states 
> - I think CONN_STATE_CHECK_REQUEST_LINE_READABLE. That is fine for a HTTP/1 
> connection as it always is between requests here. But HTTP/2 would also like 
> to use read events, but unloading a child by closing such connections is not 
> the right thing to do.
> 
>  Is there a way mod_http2 can achieve this or do we need another flag in 
> conn_rec/cs?

Does the CONN_STATE_WRITE_COMPLETION state fit this case?

The WRITE in the above name is a bit of a misnomer, because when mod_ssl 
returns WANTS_READ we return to the CONN_STATE_WRITE_COMPLETION state with the 
sense set to READ. This tells the MPM to block waiting until we’re readable, as 
opposed to block waiting until we’re writable.

> Observation 2: controlled close
>  HTTP/2 would like to send out a last frame when closing a connection in a 
> controlled/timeout way. Do we need another hook for that? Something like 
> "pre_close_connection”?

I’m assuming we would like a module to have control over the sending of that 
frame, if so, it makes sense yes.

> Observation 3: timeouts
>  a. If admins would like to have another Timeout for HTTP/2 connections, they 
> are currently stuck with configuring one Timeout which goes to 
> server_rec->timeout, right? For idle connections, clients already try to keep 
> it open longer. It would be nice to give admins a choice here for different 
> values based on the selected protocol. How to best do this?

Perhaps just as we can override the sense of the connection before returning, 
we might be able to modify the timeout before returning.

>  b. Handling own timeouts: if I want, during processing the connection, a 
> different socket timeout, I can just set it using apr_socket_timeout_set()? 
> Shall I restore it back to server->timeout before returning?

That feels wrong, ideally we want to be formally telling the MPM “give us this 
timeout” rather than doing something sneaky behind it’s back.

>  c. What would be the best way to wait for a file description *and* 
> conditional event simultaneously? mod_http2 has now a sort of pool loop with 
> double backup timer for situations where it needs to watch both client and 
> backend processes.

I’m not sure how portable it is watching for file descriptors. I have used 
libev to do this in the past and it was quite involved. However - if we can 
support this it would be really useful.

Regards,
Graham
—

async mpms + http2

[Stefan Eissing]

I just committed a change in mod_http2 that will return idle connections back 
to the mpm, expecting to be invoked again when new data arrives. I learned 
quite something about event (have not looked much at motorz yet) and am 
thinking about improvements. If knowledgeable people could give me a hand here, 
I'd highly appreciate it.

Observation 1: mapping conn_states
  As I understand it, the connection state model, as defined in conn_state_e, 
enables asynchronous processing in two states:
  a. CONN_STATE_CHECK_REQUEST_LINE_READABLE, check read, invoke 
ap_run_process_connection() on data
  b. CONN_STATE_WRITE_COMPLETION, check write, if data is still in out filters

  An invocation of ap_run_process_connection() is expected to return, usually 
in WRITE_COMPLETION state. Then any data is attempted to be written. When this 
succeeds, the next state depends on secondary flags such as c->aborted, 
c->keepalive and c->data_in_input_filters.

  HTTP/2 processing has one state where it fits into this model: when no more 
streams are open. This is what is currently implemented in trunk. 

  However, there is another state in HTTP/2 processing, where a BLOCK_READ is 
performed: flow control on streaming out responses. Basically, the client has a 
window of n bytes, the server sends n response bytes and then needs to wait for 
the client to increase the window again. This would be a good time to resume 
processing back to the mpm, and initially I did so. But that breaks under load.

  Under load, the event mpm feels free to close connections in certain states - 
I think CONN_STATE_CHECK_REQUEST_LINE_READABLE. That is fine for a HTTP/1 
connection as it always is between requests here. But HTTP/2 would also like to 
use read events, but unloading a child by closing such connections is not the 
right thing to do.

  Is there a way mod_http2 can achieve this or do we need another flag in 
conn_rec/cs?

Observation 2: controlled close
  HTTP/2 would like to send out a last frame when closing a connection in a 
controlled/timeout way. Do we need another hook for that? Something like 
"pre_close_connection"?
 
Observation 3: timeouts
  a. If admins would like to have another Timeout for HTTP/2 connections, they 
are currently stuck with configuring one Timeout which goes to 
server_rec->timeout, right? For idle connections, clients already try to keep 
it open longer. It would be nice to give admins a choice here for different 
values based on the selected protocol. How to best do this?

  b. Handling own timeouts: if I want, during processing the connection, a 
different socket timeout, I can just set it using apr_socket_timeout_set()? 
Shall I restore it back to server->timeout before returning?

  c. What would be the best way to wait for a file description *and* 
conditional event simultaneously? mod_http2 has now a sort of pool loop with 
double backup timer for situations where it needs to watch both client and 
backend processes.


Cheers,

  Stefan