Hi Folks, I'm seeing crashing in qpid proton (qpid-proton-c-devel-0.9-3 on fc22) when running my publisher at a decent rate (1000 msg/s) for a minute or so.
Does something like this look familiar? Somehow got into a state where pn_data_grow is called, but the data node coming back from it points to an invalid memory location. (gdb) bt [31/1873] #0 0x0000000006bf55df in pn_data_new (data=data@entry=0x5e6a090) at /usr/src/debug/qpid-proton-0.9/proton-c/src/codec/codec.c:1126 #1 0x0000000006bf5de8 in pn_data_add (data=data@entry=0x5e6a090) at /usr/src/debug/qpid-proton-0.9/proton-c/src/codec/codec.c:1397 #2 0x0000000006bf5fa8 in pn_data_put_ubyte (data=data@entry=0x5e6a090, ub=<optimized out>) at /usr/src/debug/qpid-proton-0.9/proton-c/src/codec/codec.c:1485 #3 0x00000000069c4c14 in qpidBridgeMsgCodec_pack (bridgeMessage=0x5e61fb0, target=target@entry=0x5e6b2c0, protonMessage=protonMessage@entry =0xffefff528) at mama/c_cpp/src/c/bridge/qpid/codec.c:119 #4 0x00000000069c5fd2 in qpidBridgePublisherImpl_enqueueMessageForAddress (msg=msg@entry=0x5e6b2c0, url=<optimized out>, impl=impl@entry=0x5e61f30) at mama/c_cpp/src/c/bridge/qpid/publisher.c:629 #5 0x00000000069c63cd in qpidBridgeMamaPublisher_send (publisher=0x5e61f30, msg=0x5e6b2c0) at mama/c_cpp/src/c/bridge/qpid/publisher.c:295 #6 0x0000000004e6be97 in mamaPublisher_send (publisher=0x5e61dc0, msg=msg@entry=0x5e6b2c0) at mama/c_cpp/src/c/publisher.c:258 #7 0x000000000040577d in publishMessageRdtsc (pubIndex=<optimized out>, msgSample=<optimized out>, nowTsc=<optimized out>) at mama/c_cpp/src/testtools/performance/c/mamaproducerc_v2.c:1620 #8 0x0000000000403b20 in main (argc=<optimized out>, argv=<optimized out>) at mama/c_cpp/src/testtools/performance/c/mamaproducerc_v2.c:782 (gdb) l 1121 { 1122 if (data->capacity <= data->size) { 1123 pn_data_grow(data); 1124 } 1125 pni_node_t *node = pn_data_node(data, ++(data->size)); 1126 node->next = 0; 1127 node->down = 0; 1128 node->children = 0; 1129 return node; 1130 } (gdb) print *data $11 = {nodes = 0x0, buf = 0x5e6a5d0, decoder = 0x5e6a6c0, encoder = 0x5e6a790, error = 0x5e6a850, str = 0x5e6a8c0, capacity = 0, size = 32769, parent = 1, current = 0, base_parent = 0, base_current = 0} (gdb) print node $12 = (pni_node_t *) 0x240000 <---- invalid memory location I ran it through valgrind as well and couldn't see any invalid memory trouncing leading up to the event: ==2332== Invalid write of size 2 ==2332== at 0x6BF55DF: pn_data_new (codec.c:1126) ==2332== by 0x6BF5DE7: pn_data_add (codec.c:1397) ==2332== by 0x6BF5FA7: pn_data_put_ubyte (codec.c:1485) ==2332== by 0x69C4C13: qpidBridgeMsgCodec_pack (codec.c:119) ==2332== by 0x69C5FD1: qpidBridgePublisherImpl_enqueueMessageForAddress (publisher.c:629) ==2332== by 0x69C63CC: qpidBridgeMamaPublisher_send (publisher.c:295) ==2332== by 0x4E6BE96: mamaPublisher_send (publisher.c:258) ==2332== by 0x40577C: publishMessageRdtsc (mamaproducerc_v2.c:1620) ==2332== by 0x403B1F: main (mamaproducerc_v2.c:782) ==2332== Address 0x240034 is not stack'd, malloc'd or (recently) free'd ==2332== ==2332== ==2332== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==2332== Access not within mapped region at address 0x240034 ==2332== at 0x6BF55DF: pn_data_new (codec.c:1126) ==2332== by 0x6BF5DE7: pn_data_add (codec.c:1397) ==2332== by 0x6BF5FA7: pn_data_put_ubyte (codec.c:1485) ==2332== by 0x69C4C13: qpidBridgeMsgCodec_pack (codec.c:119) ==2332== by 0x69C5FD1: qpidBridgePublisherImpl_enqueueMessageForAddress (publisher.c:629) ==2332== by 0x69C63CC: qpidBridgeMamaPublisher_send (publisher.c:295) ==2332== by 0x4E6BE96: mamaPublisher_send (publisher.c:258) ==2332== by 0x40577C: publishMessageRdtsc (mamaproducerc_v2.c:1620) ==2332== by 0x403B1F: main (mamaproducerc_v2.c:782) ==2332== If you believe this happened as a result of a stack ==2332== overflow in your program's main thread (unlikely but ==2332== possible), you can try to increase the size of the ==2332== main thread stack using the --main-stacksize= flag. ==2332== The main thread stack size used in this run was 8388608. Cheers, Frank