On Fri, Nov 20, 2020 at 06:08:18PM +0100, Yann Ylavic wrote: > I tried to reproduce this failure locally (debian buster) with multiple > tries but couldn't. > I think I'm using the same environment (apr-1.7.0, apr-util-1.6.1, > APR_POOL_DEBUG, docker.io/redis:latest), and in the same way running: > $ (export SSL_SESSCACHE=redis:localhost:6379; ./t/TEST -sslproto TLSv1.2 > -defines TEST_SSL_SESSCACHE t/ssl)
There was a thread on this discussing whether apr_pool_find() was really safe... I think that's the right setup, but it did not *reliably* fail just by running t/TEST alone. IIRC it did reliably fail by starting the server as in your command above, and then running "ab" against the server: ab -qd -n 4000 -c 20 -f TLS1.2 https://localhost:8532/ and https://github.com/apache/httpd/pull/140 showed this failure was specific to pool-debug. I could reliably reproduce it locally as well. > Since session caching is not a failure point in mod_ssl, I also verified > with redis up or down (and checked traffic with tcpdump), no luck either. > Does it happen only with socache_redis? > These seem to be the two threads racing: I don't think it is specific to redis, there was a similar failure in: https://travis-ci.org/github/apache/httpd/jobs/723103332 which used memcache for example. That said it's interesting and maybe significant if it doesn't fail with shmcb? Regards, Joe > > Thread 41 (Thread 0x7f59de0c7700 (LWP 32462)): > #0 0x00007f5a19215b39 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007f5a1921ac27 in free () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007f5a197b5268 in pool_clear_debug (pool=pool@entry=0x55a892d7d9b0, > file_line=<optimized out>) at memory/unix/apr_pools.c:1853 > #3 0x00007f5a197b4fee in pool_destroy_debug (pool=0x55a892d7d9b0, > file_line=<optimized out>) at memory/unix/apr_pools.c:1915 > #4 0x00007f5a197b585c in apr_pool_destroy_debug (pool=<optimized out>, > file_line=<optimized out>) at memory/unix/apr_pools.c:1957 > #5 0x000055a8929613c5 in eor_bucket_destroy (data=0x55a8931b87d8) at > eor_bucket.c:108 > #6 0x000055a89296230b in send_brigade_nonblocking (c=0x55a892d84a20, > ctx=0x55a8930b5a30, bb=0x55a892f72ac0, s=0x55a892d7d430) at > core_filters.c:570 > #7 ap_core_output_filter (f=0x55a892f3d3c0, bb=0x55a892f72ac0) at > core_filters.c:407 > #8 0x00007f5a03e76098 in bio_filter_out_pass > (outctx=outctx@entry=0x55a892f747a0) > at ssl_engine_io.c:139 > #9 0x00007f5a03e799e4 in ssl_io_filter_output (f=0x55a892f746e0, > bb=0x55a892f711a0) at ssl_engine_io.c:1988 > #10 0x00007f5a03e767a8 in ssl_io_filter_coalesce (f=0x55a892f72c40, > bb=0x55a892f711a0) at ssl_engine_io.c:1922 > #11 0x000055a89292dc6e in ap_process_request_after_handler (r=<optimized > out>) at http_request.c:375 > #12 0x000055a89292b731 in ap_process_http_async_connection > (c=0x55a892d84a20) at http_core.c:158 > #13 ap_process_http_connection (c=0x55a892d84a20) at http_core.c:252 > #14 0x000055a892944d30 in ap_run_process_connection (c=c@entry=0x55a892d84a20) > at connection.c:42 > #15 0x00007f5a17b8c65d in process_socket (thd=0x55a89313fcc0, p=<optimized > out>, sock=<optimized out>, cs=0x55a892d0d650, my_child_num=<optimized > out>, my_thread_num=<optimized out>) at event.c:1094 > #16 0x00007f5a17b8cf8f in worker_thread (thd=0x55a89313fcc0, > dummy=<optimized out>) at event.c:2383 > #17 0x00007f5a1957b6db in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #18 0x00007f5a192a488f in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 66 (Thread 0x7f59df0c9700 (LWP 32460)): > #0 0x00007f5a191c2187 in kill () from /lib/x86_64-linux-gnu/libc.so.6 > #1 <signal handler called> > #2 0x00007f5a191c1e97 in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x00007f5a191c3801 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #4 0x00007f5a197adf62 in apr_table_mergen (t=0x55a89306dfb0, > key=key@entry=0x55a8929870aa > "Connection", val=val@entry=0x55a89297ff86 "close") at > tables/apr_tables.c:750 > #5 0x000055a89292bfff in ap_set_keepalive (r=r@entry=0x55a892fd2540) at > http_protocol.c:309 > #6 0x000055a892931435 in ap_http_header_filter (f=0x55a892f39c30, > b=0x55a892f54ea0) at http_filters.c:1372 > #7 0x000055a89294f071 in ap_content_length_filter (f=0x55a892f38140, > b=0x55a892f54ea0) at protocol.c:2040 > #8 0x000055a8929332a7 in ap_byterange_filter (f=0x55a892d22370, > bb=0x55a892f54ea0) at byterange_filter.c:463 > #9 0x00007f5a04ab8e14 in session_output_filter (f=0x55a892f3d8b0, > in=0x55a892f54ea0) at mod_session.c:501 > #10 0x00007f5a05efdc10 in ap_proxy_http_process_response > (req=req@entry=0x55a892f46bf0) > at mod_proxy_http.c:1997 > #11 0x00007f5a05f0018f in proxy_http_handler (r=<optimized out>, > worker=<optimized out>, conf=0x55a892e614f0, url=<optimized out>, > proxyname=0x0, proxyport=<optimized out>) at mod_proxy_http.c:2326 > #12 0x00007f5a0651f7ec in proxy_run_scheme_handler (r=r@entry=0x55a892fd2540, > worker=0x55a892e61780, conf=conf@entry=0x55a892e614f0, url=0x55a89325ea96 " > https://localhost:8532/";, proxyhost=proxyhost@entry=0x0, > proxyport=proxyport@entry=0) at mod_proxy.c:3519 > #13 0x00007f5a0652189e in proxy_handler (r=0x55a892fd2540) at > mod_proxy.c:1525 > #14 0x000055a892935b00 in ap_run_handler (r=r@entry=0x55a892fd2540) at > config.c:170 > #15 0x000055a89293609d in ap_invoke_handler (r=r@entry=0x55a892fd2540) at > config.c:444 > #16 0x000055a89292ec9b in ap_process_async_request (r=0x55a892fd2540) at > http_request.c:463 > #17 0x000055a89292b731 in ap_process_http_async_connection > (c=0x55a892d13490) at http_core.c:158 > #18 ap_process_http_connection (c=0x55a892d13490) at http_core.c:252 > #19 0x000055a892944d30 in ap_run_process_connection (c=c@entry=0x55a892d13490) > at connection.c:42 > #20 0x00007f5a17b8c65d in process_socket (thd=0x55a8931410b0, p=<optimized > out>, sock=<optimized out>, cs=0x55a892cfe760, my_child_num=<optimized > out>, my_thread_num=<optimized out>) at event.c:1094 > #21 0x00007f5a17b8cf8f in worker_thread (thd=0x55a8931410b0, > dummy=<optimized out>) at event.c:2383 > #22 0x00007f5a1957b6db in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #23 0x00007f5a192a488f in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > But we don't in Thread #41 whether the request is the same as in Thread #66 > (0x55a892fd2540).. > > Any idea on how to reproduce it? I'm out of ideas currently :/ > > Regards; > Yann. > > On Thu, Nov 19, 2020 at 1:34 PM Travis CI <[email protected]> wrote: > > > > apache / httpd > > > > trunk > > > > Build #1216 was broken > > 19 mins and 3 secs > > Yann Ylavic > > 2865f25 CHANGESET → > > > > Remove indentation preserved in r1883639 to ease review. > > > > No functional changes. > > > > > > git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883640 > 13f79535-47bb-0310-9956-ffa450edef68 > > > > Want to know about upcoming build environment updates? > > > > Would you like to stay up-to-date with the upcoming Travis CI build > environment updates? We set up a mailing list for you! > > > > SIGN UP HERE > > Documentation about Travis CI > > Have any questions? We're here to help. > > Unsubscribe from build emails from the apache/httpd repository. > > To unsubscribe from all build emails, please update your settings. > > > > > On Thu, Nov 19, 2020 at 1:34 PM Travis CI <[email protected]> wrote: > > > apache > > > > / > > > > httpd > > > > <https://travis-ci.com/github/apache/httpd?utm_medium=notification&utm_source=email> > > > > [image: branch icon]trunk <https://github.com/apache/httpd/tree/trunk> > > [image: build has failed] > > Build #1216 was broken > > <https://travis-ci.com/github/apache/httpd/builds/202714023?utm_medium=notification&utm_source=email> > > [image: arrow to build time] > > [image: clock icon]19 mins and 3 secs > > > > [image: Yann Ylavic avatar]Yann Ylavic > > 2865f25 CHANGESET → > > <https://github.com/apache/httpd/compare/7c220fc67527...2865f2584b4d> > > > > Remove indentation preserved in r1883639 to ease review. > > > > No functional changes. > > > > > > git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883640 > > 13f79535-47bb-0310-9956-ffa450edef68 > > > > Want to know about upcoming build environment updates? > > > > Would you like to stay up-to-date with the upcoming Travis CI build > > environment updates? We set up a mailing list for you! > > SIGN UP HERE <http://eepurl.com/9OCsP> > > > > [image: book icon] > > > > Documentation <https://docs.travis-ci.com/> about Travis CI > > Have any questions? We're here to help. <[email protected]> > > Unsubscribe > > <https://travis-ci.com/account/preferences/unsubscribe?repository=16806660&utm_medium=notification&utm_source=email> > > from build emails from the apache/httpd repository. > > To unsubscribe from *all* build emails, please update your settings > > <https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email>. > > > > [image: black and white travis ci logo] <https://travis-ci.com> > > > > Travis CI GmbH, Rigaer Str. 8, 10427 Berlin, Germany | GF/CEO: Randy > > Jacops | Contact: [email protected] | Amtsgericht Charlottenburg, > > Berlin, HRB 140133 B | Umsatzsteuer-ID gemäß §27 a Umsatzsteuergesetz: > > DE282002648 > >
