Re: proton crash in pn_messenger_free()

[Michael Ivanov]

Hallo,

I have investigated the crash a bit further. I have added decref/incref
trace printout and I got the following at the moment of crash:

COLLECTOR 0xba9050 FREE
COLLECTOR 0xba9050 RELEASE
COLLECTOR 0xba9050 DRAIN
COLLECTOR 0xba9050 POP: event 0xbcdd10/pn_link
PN_OBJ_DECREF: object 0xbcdd10 class [pn_event] refcount 1
PN_OBJ_DECREF: object 0xbd68c0 class [pn_link] refcount 1
PN_OBJ_INCREF: object 0xbd68c0 class [pn_link] refcount 0
PN_OBJ_DECREF: object 0xbd57f0 class [pn_session] refcount 1
PN_OBJ_INCREF: object 0xbd57f0 class [pn_session] refcount 0
PN_OBJ_DECREF: object 0xbaf370 class [pn_connection] refcount 3
PN_OBJ_INCREF: object 0xbcdd10 class [pn_event] refcount 0
PN_OBJ_DECREF: object 0xba9090 class [pn_list] refcount 5
COLLECTOR 0xba9050 POP: event 0xbd0940/pn_transport
PN_OBJ_DECREF: object 0xbd0940 class [pn_event] refcount 1
PN_OBJ_DECREF: object 0xbc1430 class [pn_transport] refcount 1
PN_OBJ_INCREF: object 0xbc1430 class [pn_transport] refcount 0
PN_OBJ_DECREF: object 0xbaf370 class [pn_connection] refcount 2
PN_OBJ_INCREF: object 0xbd0940 class [pn_event] refcount 0
PN_OBJ_DECREF: object 0xba9090 class [pn_list] refcount 4
COLLECTOR 0xba9050 POP: event 0xbcddb0/pn_connection
PN_OBJ_DECREF: object 0xbcddb0 class [pn_event] refcount 1
PN_OBJ_DECREF: object 0xbaf370 class [pn_connection] refcount 1
PN_OBJ_DECREF: object 0xbc1430 class [pn_transport] refcount 1
PN_OBJ_DECREF: object 0xbc25b0 class [pn_record] refcount 1
PN_OBJ_DECREF: object 0xbc2600 class [pn_data] refcount 1
PN_OBJ_DECREF: object 0xbc27c0 class [pn_string] refcount 1
PN_OBJ_DECREF: object 0xbc26e0 class [pn_decoder] refcount 1
PN_OBJ_DECREF: object 0xbc2740 class [pn_encoder] refcount 1
PN_OBJ_DECREF: object 0xbc2810 class [pn_data] refcount 1
. . . . . . . . . . . . . .
PN_OBJ_DECREF: object 0xbda230 class [pn_string] refcount 1
PN_OBJ_DECREF: object 0xbda150 class [pn_decoder] refcount 1
PN_OBJ_DECREF: object 0xbda1b0 class [pn_encoder] refcount 1
PN_OBJ_DECREF: object 0xbda020 class [pn_string] refcount 1
PN_OBJ_DECREF: object 0xbd9fd0 class [pn_string] refcount 1
PN_OBJ_INCREF: object 0xbcddb0 class [pn_event] refcount 0
PN_OBJ_DECREF: object 0xba9090 class [pn_list] refcount 3
COLLECTOR 0xba9050 POP: event 0xbcde50/pn_transport
PN_OBJ_DECREF: object 0xbcde50 class [pn_event] refcount 1
PN_OBJ_DECREF: object 0xbc1430 class [pn_transport] refcount 0
evtst: 
/home/ivans/misc/mq/qpid-proton-0.10.fix/proton-c/src/object/object.c:244: 
pn_object_decref: Assertion `head->refcount >
0' failed.

Transport object 0xbc1430 seems to have been deleted when event 0xbcddb0
for pn_connection was destroyed. But next event popped for the collector
refers to the same transport (0xbc1430, already destroyed).

Any suggestions how to fix this or speed the investigation up?

best regards,
-- 
 \   / |   |
 (OvO) |  Mikhail Iwanow   |
 (^^^) |   |
  \^/  |  E-mail:  iv...@logit-ag.de   |
  ^ ^  |   |

Re: proton crash in pn_messenger_free()

[Michael Ivanov]

It's still the same in proton 0.10 The line numbers are a bit different,
but the error seems to be the same:

==6040== Invalid read of size 8
==6040==at 0x53679F5: pn_object_reify (object.c:216)
==6040==by 0x5367BA6: pn_class_decref (object.c:92)
==6040==by 0x5375BBF: pn_event_finalize_cast (event.c:211)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375D71: pn_collector_pop (event.c:188)
==6040==by 0x5375DC7: pn_collector_release (event.c:55)
==6040==by 0x5375DE8: pn_collector_free (event.c:108)
==6040==by 0x53816E9: pn_messenger_free (messenger.c:828)
==6040==by 0x510B1C1: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==6040==by 0x51190A7: kit::Application::~Application() (kit+.cc:172)
==6040==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==6040==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==6040==by 0x40B755: main (cplc.cc:214)
==6040==  Address 0xdc68e50 is 0 bytes inside a block of size 416 free'd
==6040==at 0x4C29577: free (in 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==6040==by 0x5367BE5: pn_class_decref (object.c:103)
==6040==by 0x5372D14: pn_connection_finalize (engine.c:468)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375BBF: pn_event_finalize_cast (event.c:211)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375D71: pn_collector_pop (event.c:188)
==6040==by 0x5375DC7: pn_collector_release (event.c:55)
==6040==by 0x5375DE8: pn_collector_free (event.c:108)
==6040==by 0x53816E9: pn_messenger_free (messenger.c:828)
==6040==by 0x510B1C1: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==6040==by 0x51190A7: kit::Application::~Application() (kit+.cc:172)
==6040==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==6040==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==6040==by 0x40B755: main (cplc.cc:214)
==6040==
==6040== Invalid read of size 4
==6040==at 0x5367A30: pn_object_decref (object.c:239)
==6040==by 0x5367BAF: pn_class_decref (object.c:93)
==6040==by 0x5375BBF: pn_event_finalize_cast (event.c:211)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375D71: pn_collector_pop (event.c:188)
==6040==by 0x5375DC7: pn_collector_release (event.c:55)
==6040==by 0x5375DE8: pn_collector_free (event.c:108)
==6040==by 0x53816E9: pn_messenger_free (messenger.c:828)
==6040==by 0x510B1C1: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==6040==by 0x51190A7: kit::Application::~Application() (kit+.cc:172)
==6040==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==6040==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==6040==by 0x40B755: main (cplc.cc:214)
==6040==  Address 0xdc68e58 is 8 bytes inside a block of size 416 free'd
==6040==at 0x4C29577: free (in 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==6040==by 0x5367BE5: pn_class_decref (object.c:103)
==6040==by 0x5372D14: pn_connection_finalize (engine.c:468)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375BBF: pn_event_finalize_cast (event.c:211)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375D71: pn_collector_pop (event.c:188)
==6040==by 0x5375DC7: pn_collector_release (event.c:55)
==6040==by 0x5375DE8: pn_collector_free (event.c:108)
==6040==by 0x53816E9: pn_messenger_free (messenger.c:828)
==6040==by 0x510B1C1: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==6040==by 0x51190A7: kit::Application::~Application() (kit+.cc:172)
==6040==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==6040==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==6040==by 0x40B755: main (cplc.cc:214)
==6040==
==6040== Invalid read of size 4
==6040==at 0x5367A20: pn_object_refcount (object.c:232)
==6040==by 0x5367BB5: pn_class_decref (object.c:94)
==6040==by 0x5375BBF: pn_event_finalize_cast (event.c:211)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375D71: pn_collector_pop (event.c:188)
==6040==by 0x5375DC7: pn_collector_release (event.c:55)
==6040==by 0x5375DE8: pn_collector_free (event.c:108)
==6040==by 0x53816E9: pn_messenger_free (messenger.c:828)
==6040==by 0x510B1C1: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==6040==by 0x51190A7: kit::Application::~Application() (kit+.cc:172)
==6040==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==6040==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==6040==by 0x40B755: main (cplc.cc:214)
==6040==  Address 0xdc68e58 is 8 bytes inside a block of size 416 free'd
==6040==at 0x4C29577: free (in 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==6040==by 0x5367BE5: pn_class_decref (object.c:103)
==6040==by 0x5372D14: pn_connection_finalize (engine.c:468)
==6040==by 0x5367BC7: pn_class_decref (object.c:97)
==6040==by 0x5375BBF: pn_event_finalize_cast (event.c:211)
==6040==by 

proton crash in pn_messenger_free()

[Michael Ivanov]

Hallo,

I am using proton 0.91 and I constantly experience crashes on process 
termination.
I use two messengers, one to read primary input queue and the other to send 
replies.
Primary messenger is used in non-blocking mode. On termination I do the 
following:

   if (_reply) {
  pn_messenger_stop(_reply);
  pn_messenger_free(_reply);
   }
   if (_primary) {
  pn_messenger_set_blocking(_primary, 1);
  pn_messenger_stop(_primary);
  pn_messenger_free(_primary); /// <
   }

When prmary messenger is freed the process almost always crashes. I have 
investigated
this using valgrind and I got the following:

==14853== Invalid read of size 8
==14853==at 0x5368435: pn_object_reify (object.c:213)
==14853==by 0x53685D6: pn_class_decref (object.c:92)
==14853==by 0x537634F: pn_event_finalize_cast (event.c:190)
==14853==by 0x53685F7: pn_class_decref (object.c:97)
==14853==by 0x53764F1: pn_collector_pop (event.c:167)
==14853==by 0x5376547: pn_collector_release (event.c:34)
==14853==by 0x5376568: pn_collector_free (event.c:87)
==14853==by 0x5382539: pn_messenger_free (messenger.c:780)
==14853==by 0x510AB39: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==14853==by 0x5118DAB: kit::Application::~Application() (kit+.cc:172)
==14853==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==14853==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==14853==by 0x40B755: main (cplc.cc:215)
==14853==  Address 0xd79c8e0 is 0 bytes inside a block of size 408 free'd
==14853==at 0x4C29577: free (in 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14853==by 0x5368615: pn_class_decref (object.c:103)
==14853==by 0x5373784: pn_connection_finalize (engine.c:463)
==14853==by 0x53685F7: pn_class_decref (object.c:97)
==14853==by 0x537634F: pn_event_finalize_cast (event.c:190)
==14853==by 0x53685F7: pn_class_decref (object.c:97)
==14853==by 0x53764F1: pn_collector_pop (event.c:167)
==14853==by 0x5376547: pn_collector_release (event.c:34)
==14853==by 0x5376568: pn_collector_free (event.c:87)
==14853==by 0x5382539: pn_messenger_free (messenger.c:780)
==14853==by 0x510AB39: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==14853==by 0x5118DAB: kit::Application::~Application() (kit+.cc:172)
==14853==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==14853==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==14853==by 0x40B755: main (cplc.cc:215)
==14853==
==14853== Invalid read of size 4
==14853==at 0x5368470: pn_object_decref (object.c:236)
==14853==by 0x53685DF: pn_class_decref (object.c:93)
==14853==by 0x537634F: pn_event_finalize_cast (event.c:190)
==14853==by 0x53685F7: pn_class_decref (object.c:97)
==14853==by 0x53764F1: pn_collector_pop (event.c:167)
==14853==by 0x5376547: pn_collector_release (event.c:34)
==14853==by 0x5376568: pn_collector_free (event.c:87)
==14853==by 0x5382539: pn_messenger_free (messenger.c:780)
==14853==by 0x510AB39: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==14853==by 0x5118DAB: kit::Application::~Application() (kit+.cc:172)
==14853==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==14853==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==14853==by 0x40B755: main (cplc.cc:215)
==14853==  Address 0xd79c8e8 is 8 bytes inside a block of size 408 free'd
==14853==at 0x4C29577: free (in 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14853==by 0x5368615: pn_class_decref (object.c:103)
==14853==by 0x5373784: pn_connection_finalize (engine.c:463)
==14853==by 0x53685F7: pn_class_decref (object.c:97)
==14853==by 0x537634F: pn_event_finalize_cast (event.c:190)
==14853==by 0x53685F7: pn_class_decref (object.c:97)
==14853==by 0x53764F1: pn_collector_pop (event.c:167)
==14853==by 0x5376547: pn_collector_release (event.c:34)
==14853==by 0x5376568: pn_collector_free (event.c:87)
==14853==by 0x5382539: pn_messenger_free (messenger.c:780)
==14853==by 0x510AB39: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==14853==by 0x5118DAB: kit::Application::~Application() (kit+.cc:172)
==14853==by 0x416F42: plc::Driver::~Driver() (plc-driver.cc:127)
==14853==by 0x40AE5E: CPLC::~CPLC() (cplc.cc:40)
==14853==by 0x40B755: main (cplc.cc:215)
==14853==
==14853== Invalid read of size 4
==14853==at 0x5368460: pn_object_refcount (object.c:229)
==14853==by 0x53685E5: pn_class_decref (object.c:94)
==14853==by 0x537634F: pn_event_finalize_cast (event.c:190)
==14853==by 0x53685F7: pn_class_decref (object.c:97)
==14853==by 0x53764F1: pn_collector_pop (event.c:167)
==14853==by 0x5376547: pn_collector_release (event.c:34)
==14853==by 0x5376568: pn_collector_free (event.c:87)
==14853==by 0x5382539: pn_messenger_free (messenger.c:780)
==14853==by 0x510AB39: kit::EventLoop::~EventLoop() (EventLoop.cc:84)
==14853==by 0x5118DAB: kit::Application::~Application()