[
https://issues.apache.org/jira/browse/PROTON-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14051826#comment-14051826
]
michael goulish commented on PROTON-625:
----------------------------------------
Here's what happens, and a fix.
1. pni_map_entry() calls pni_map_ensure() to make sure map
has enough capacity.
2. The capacity-increasing loop in pni_map_ensure() has two
conditions on it: increase the capacity if map->capacity
is too small, or if map 'load' is greater than map->load_factor.
( Map load is ... meaning not obvious to me. )
3. If pni_map_ensure() returns true, then pni_map_entry() will
call itself recursively, and keep doing that until
pni_map_ensure() returns false.
'False' means 'I made no change.'
4. But it is possible for pni_map_ensure() to make no change,
and yet return true.
Here is how it happened in my most recent test:
map->capacity 512
capacity 331
pni_map_load(map) 0.750000
map->load_factor 0.750000
5. Those values made *both* conditions on the capacity-
increasing loop in pni_map_ensure() false.
So it didn't do anything to change the map.
But it returned true.
So pni_map_entry() called itself.
But nothing had changed.
And away we go.
FIX ------------------------------------
Make the test on the if at the top of pni_map_ensure
say this:
if (capacity <= map->capacity && load <= map->load_factor) {
( Added '=' to the load test. )
After that, I ran twenty tests with no failure.
Previously, failure probability on my system
was 0.3. So odds of 20 in a row happening
by chance is a little less than 1 in 1000.
> Biggest Backtrace Ever!
> -----------------------
>
> Key: PROTON-625
> URL: https://issues.apache.org/jira/browse/PROTON-625
> Project: Qpid Proton
> Issue Type: Bug
> Components: proton-c
> Affects Versions: 0.8
> Reporter: michael goulish
>
> I am saving all my stuff so I can repro on demand.
> It doesn't happen every time, but it's about 50%.
> ------------------------------------------
> On one box, I have a dispatch router.
> On the other box, I have 10 clients: 5 Messenger-based receivers, and 5
> qpid-messaging-based senders.
> Each client will handle 100 addresses, of the form "mick/0" ... "mick/1" ...
> & c.
> 100 messages will be sent to each address.
> I start the 5 receivers first. They start OK. Dispatch router happy &
> stable.
> Wait a few seconds.
> I start the 5 senders, from a bash script.
> The first sender is already sending when the 2nd, 3rd, 4th start.
> After a few of them start,but before all have finished starting, a few
> seconds into the script, the crash occurs. ( If they all start up
> successfully, no crash. )
> The crash occurs in the dispatch router.
> Here is the biggest backtrace ever:
> #0 0x0000003cf9879ad1 in _int_malloc (av=0x7f101c000020, bytes=16384) at
> malloc.c:4383
> #1 0x0000003cf987a911 in __libc_malloc (bytes=16384) at malloc.c:3664
> #2 0x00000039c6c1650a in pni_map_allocate () from
> /usr/lib64/libqpid-proton.so.2
> #3 0x00000039c6c16a3a in pni_map_ensure () from
> /usr/lib64/libqpid-proton.so.2
> #4 0x00000039c6c16c45 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #5 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #6 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #7 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #8 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #9 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #10 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #11 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #12 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #13 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> #14 0x00000039c6c16c64 in pni_map_entry () from /usr/lib64/libqpid-proton.so.2
> .
> .
> .
> .
> #93549 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93550 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93551 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93552 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93553 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93554 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93555 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93556 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93557 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93558 0x00000039c6c16c64 in pni_map_entry () from
> /usr/lib64/libqpid-proton.so.2
> #93559 0x00000039c6c16dc0 in pn_map_put () from /usr/lib64/libqpid-proton.so.2
> #93560 0x00000039c6c17226 in pn_hash_put () from
> /usr/lib64/libqpid-proton.so.2
> #93561 0x00000039c6c2a643 in pn_delivery_map_push () from
> /usr/lib64/libqpid-proton.so.2
> #93562 0x00000039c6c2c44b in pn_do_transfer () from
> /usr/lib64/libqpid-proton.so.2
> #93563 0x00000039c6c24385 in pn_dispatch_frame () from
> /usr/lib64/libqpid-proton.so.2
> #93564 0x00000039c6c2448f in pn_dispatcher_input () from
> /usr/lib64/libqpid-proton.so.2
> #93565 0x00000039c6c2d68b in pn_input_read_amqp () from
> /usr/lib64/libqpid-proton.so.2
> #93566 0x00000039c6c3011a in pn_io_layer_input_passthru () from
> /usr/lib64/libqpid-proton.so.2
> #93567 0x00000039c6c3011a in pn_io_layer_input_passthru () from
> /usr/lib64/libqpid-proton.so.2
> #93568 0x00000039c6c2d275 in transport_consume () from
> /usr/lib64/libqpid-proton.so.2
> #93569 0x00000039c6c304cd in pn_transport_process () from
> /usr/lib64/libqpid-proton.so.2
> #93570 0x00000039c6c3e40c in pn_connector_process () from
> /usr/lib64/libqpid-proton.so.2
> #93571 0x00007f1060c60460 in process_connector () from
> /home/mick/dispatch/build/libqpid-dispatch.so.0
> #93572 0x00007f1060c61017 in thread_run () from
> /home/mick/dispatch/build/libqpid-dispatch.so.0
> #93573 0x0000003cf9c07851 in start_thread (arg=0x7f1052bfd700) at
> pthread_create.c:301
> #93574 0x0000003cf98e890d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
--
This message was sent by Atlassian JIRA
(v6.2#6252)
