On Wed, 20 Feb 2008, Niklas Edmundsson wrote:
In any case, I should probably try to figure out how to reproduce this thing.
All coredumps I've looked at have been when serving DVD images, which of
course works flawlessly when I try it...
OK, I've been able to reproduce this, and it looks really bad because:
- I'm able to reproduce without having mod_cache loaded, ie. vanilla
httpd.
- It's as easy as continuing an aborted download, so it's a trivial
DOS.
So, to reproduce I did:
1) Download 2222288895 bytes of the total 4444577792 bytes of a DVD
image (debian-31r7-i386-binary-2.iso if you're curious).
2) Continue the download by doing wget -cS http://whatever/file.iso
This coredumps the server, immediately closing the connection to the
client.
Backtrace of coredump is:
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7cefca6 in kill () from /lib/tls/i686/cmov/libc.so.6
#2 0x08089a03 in sig_coredump (sig=11) at mpm_common.c:1235
#3 <signal handler called>
#4 0x00000000 in ?? ()
#5 0x08093010 in ap_byterange_filter (f=0x81606a0, bb=0x8161360)
at byterange_filter.c:271
#6 0x0808aec5 in ap_pass_brigade (next=0x81606a0, bb=0x8161360)
at util_filter.c:526
#7 0x08077576 in default_handler (r=0x815f968) at core.c:3740
#8 0x0807df8d in ap_run_handler (r=0x815f968) at config.c:157
#9 0x0807e6d7 in ap_invoke_handler (r=0x815f968) at config.c:372
#10 0x0808ea7c in ap_process_request (r=0x815f968) at http_request.c:258
#11 0x0808b543 in ap_process_http_connection (c=0x815bb08) at http_core.c:190
#12 0x08086df3 in ap_run_process_connection (c=0x815bb08) at connection.c:43
#13 0x08087274 in ap_process_connection (c=0x815bb08, csd=0x815b958)
at connection.c:178
#14 0x08094b00 in process_socket (p=0x815b920, sock=0x815b958, my_child_num=0,
my_thread_num=0, bucket_alloc=0x815d928) at worker.c:544
#15 0x080953c8 in worker_thread (thd=0x812d378, dummy=0x815b460)
at worker.c:894
#16 0xb7e87eac in dummy_worker (opaque=0x812d378)
at threadproc/unix/thread.c:142
#17 0xb7e1846b in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#18 0xb7d9873e in clone () from /lib/tls/i686/cmov/libc.so.6
(gdb) dump_bucket ec
bucket=¨0¸(0x08161364) length=135664344 data=0x080641b0
contents=[**unknown**] rc=n/a
(gdb) print *ec
$1 = {link = {next = 0x815db00, prev = 0x8169a50}, type = 0x815d928,
length = 135664344, start = -5193905754803399840, data = 0x80641b0,
free = 0x8161390, list = 0x1}
(gdb) print *ec->type
$2 = {name = 0x815b920 "¨À\v\b0ù\025\b\030Ñ\022\b¸9\026\b\030À\025\b",
num_func = 135641240, is_metadata = APR_BUCKET_DATA, destroy = 0x816bd00,
read = 0x58, setaside = 0x815d928, split = 0x815d910, copy = 0}
(gdb) dump_brigade bb
dump of brigade 0x8161360
| type (address) | length | data addr
---------------------------------------------------
0 | FILE (0x0815db00) | 16777216 | 0x0815daa8
1 | FILE (0x0815db58) | 16777216 | 0x0815daa8
<snip>
265 | FILE (0x081699f8) | 16777216 | 0x0815daa8
266 | FILE (0x0815d948) | 15392768 | 0x0815daa8
267 | EOS (0x08169a50) | 0 | 0x00000000
end of brigade
So it looks to me that the bb brigade is intact, but the ec bucket has
been smashed into bits and pieces...
This is on ubuntu710-i386, configured with:
./configure --prefix=/tmp/2.2.8.worker.debug --with-mpm=worker
--sysconfdir=/var/conf/apache2 --with-included-apr
--enable-nonportable-atomics=yes --enable-layout=GNU --with-gdbm
--without-berkeley-db --enable-mods-shared=all --enable-cache=shared
--enable-disk-cache=shared --enable-ssl=shared --enable-cgi=shared
--enable-suexec --with-suexec-caller=yada --with-suexec-uidmin=1000
--with-suexec-gidmin=1000 CFLAGS="-march=i686 -g"
So, is anyone else able to reproduce this?
Any clue on what's the reason? I see some notes in CHANGES about
reusing brigades and so on, which might be related. However I'm way
too unclued to figure out even the general area of where things go
wrong in bucket-land...
I did some other tests, for example fetching 45809664 bytes of the
file and then continuing, I get this reply:
Content-Length: 103800832
Content-Range: bytes 45809664-4444577791/4444577792
Which is of course dead wrong, and using wget which trusts
Content-Length I end up with a truncated file. Talking to a
httpd-2.2.6 server I get the correct reply.
Something is really messed up in 2.2.8 (and I'm partly to blame, since
I didn't have time to test it prior to release ;)
An unrelated note: Why on earth chop the poor file into 267 buckets?
MAX_BUCKET_SIZE in srclib/apr-util/buckets/apr_brigade.c is 1GB (which
works, that's what I use with my DISKCACHE buckets), where does 16MB
come from?
/Nikke - keeping a brown paper bag handy, any takers?
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---------------------------------------------------------------------------
Many people are unenthusiastic about your work.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
