https://bz.apache.org/bugzilla/show_bug.cgi?id=64234
Bug ID: 64234
Summary: Fix random memory-corruption in proxy
error-reading-response path for http2 worker setup
Product: Apache httpd-2
Version: 2.4.41
Hardware: HP
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_proxy
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
Created attachment 37102
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37102&action=edit
Bugfix for above mod_proxy_http issue
Hello,
in our company we run a busy httpd service with http2-connections to the client
side and http-proxy-connections to the backend side. The httpd is configured
with 64 worker-threads per process, and the machines use redhat7 linux.
Without the attached patch, we saw random memory corruptions, mostly related to
bucket-structs which got overridden with other data or strings.
- this often lead to busy threads, because one of the many brigade-loops in the
apache could not find its sentinel and looped forever.
- in other cases this simply segfaulted an apache-thread or triggered one of
the
assertions like 'assert(b->length != (apr_size_t)-1)' in
modules/http2/h2_stream.c:add_buffered_data(),
where a bucket_type_socket appeared in a brigade where it did not belong.
What happens here?
- the proxy-http-thread runs in a different thread than the http2-server-side.
- in mod_proxy_http.c:ap_proxy_http_process_response() the response-data which
is
read from the backend-proxy-connection is stored in the temporary brigade
'bb'.
In the normal datapath the reference-count of the heap-buckets in 'bb' is
first
increased, before the transformed brigade 'pass_bb' is passed to the
output-filters
of the http2-client-connection.
mod_proxy_http.c:ap_proxy_http_process_response() the response-data which is
-> rv = ap_get_brigade(backend->r->input_filters, bb, AP_MODE_READBYTES,
...);
-> ap_proxy_buckets_lifetime_transform(r, bb, pass_bb);
-> apr_bucket_read(e, &data, &bytes, APR_BLOCK_READ);
-> increases refcount on HEAP-buckets
new = apr_bucket_transient_create(data, bytes, bucket_alloc);
-> stores buffer-pointer in simple 'transient' bucket
APR_BRIGADE_INSERT_TAIL(to, new);
ap_pass_brigade(r->output_filters, pass_bb);
->
modules/http2/h2_task:h2_filter_slave_output()->slave_out()->send_out():
-> h2_beam_send()->append_bucket()
-> creates beam-bucket around HEAP-bucket and copies it into
beam->send_list without an extra refcount.
apr_brigade_cleanup(pass_bb);
apr_brigade_cleanup(bb);
-> all HEAP-buckets with refcount 1 were freed() here
The http2 module later works from another thread on these buckets and frees
them up when finished. This is ok as long as the refcount on the HEAP-buckets
was increased correctly. a.e:
-> h2_session.c:stream_data_cb()
-> h2_stream_out_prepare()
-> h2_beam_receive()
-> while(bsender = H2_BLIST_FIRST(&beam->send_list))
APR_BUCKET_REMOVE(bsender);
[...]
- In the 'error-reading-response' path in ap_proxy_http_process_response() the
reference count of the heap-buckets in 'bb' is not increased, but the full
brigade
is still passed to the http2-thread - with the impact of write-after-free
operations.
If the memory of these heap-buckets was already allocated again, a corruption
happens.
I hope this will help.
Greetings,
Barnim
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
