On Sun, Sep 9, 2012 at 2:31 PM, Alex Bligh <a...@alex.org.uk> wrote:
> I am trying to work out how to develop a thread-safe module with two
> threads, one thread reading and one thread writing. I'm using mpm-prefork
> on apache 2.2.14-5ubuntu8.9 in case that matters. The module is a websocket
> proxy.
>
> My main thread (#1) is doing (in essence - error handling removed)
>
> apr_pool_create(&opool, r->pool);
> apr_bucket_alloc_t *oallocator = apr_bucket_alloc_create(opool);
> apr_bucket_brigade *obb = apr_brigade_create(opool, oallocator);
> apr_thread_create (... otherthread ...);
> while (1) {
> bb = apr_brigade_create(r->pool, r->connection->bucket_alloc);
> ap_get_brigade(r->input_filters, bb, AP_MODE_READBYTES,
> APR_BLOCK_READ, bufsiz));
> apr_brigade_flatten(bb, buffer, &bufsiz);
> apr_brigade_destroy(bb);
> do_something_with(buffer);
> }
>
> The other thread (#2) is doing (in essence)
>
> while (1) {
> apr_pollset_poll(recvpollset, timeout, ... );
> apr_socket_recv(socket, buf, ...)
> do_something_else_with(buf);
> ap_fwrite(r->connection->output_filters, obb, ...);
> ap_fflush(r->connection->output_filters, obb, ...);
> }
>
> I am suffering from very occasional corruption of the bucket brigade which
> normally shows up as a corrupted pool pointer or a bogus bucket entry in
> thread #1 (for instance a SEGV in apr_brigade_length). Interestingly this
> is the relatively quiet input brigade which is only ever touched by the
> main apache thread. It's almost as if an allocator is not thread safe.
That's because it isn't unless you explicitly make it so (which no MPM does).
> However, I'm using a separate bucket allocator (see code above) and (at
> least in my code) a separate pool.
They're not really separate, the sub pool is created off r->pool. You
should probably use apr_pool_create_ex() here with parent=NULL and an
allocator that you created with apr_allocator_create() +
apr_allocator_mutex_set().
That won't solve all your problems though. Bucket brigades are not
thread safe, you will need something to synchronize on.
> Could the output filter chain somehow be using the allocator attached to
> the request (i.e. thread #1)? If so, how can I stop this? I can't run the
> ap_fwrite/ap_flush under a mutex and have the same mutex held during
> ap_get_brigade, as the latter blocks (and I can't see how to use the
> non-blocking version without spinning).
>
> [apologies for the partial dupe on the apache-users mailing list - I think
> I've got nearer the problem since then and this list seems more
> appropriate]
>
> --
> Alex Bligh
