Client (bufferevent + openssl) crash *only under load situation*. Valgrind points to this backtrace. Any help is greatly appreciated ? Please let us know if you need any more information.
Thanks -Praveen Version: 2.0.22 ==695== Invalid write of size 8 ==695== at 0x52CF66B: evmap_io_add (evmap.c:328) ==695== by 0x52BC638: event_add_internal (event.c:2073) ==695== by 0x52BC150: event_add (event.c:1966) ==695== by 0x5504678: bufferevent_openssl_new_impl (bufferevent_openssl.c:1368) ==695== by 0x550482B: bufferevent_openssl_socket_new (bufferevent_openssl.c:1440) ==695== by 0x43211A: tls_connect_timer_cb (my_client_peer.c:528) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) ==695== by 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650) ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) ==695== by 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== ==695== Invalid write of size 8 ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) ==695== by 0x52BCE7B: event_del_internal (event.c:2251) ==695== by 0x52BCBD7: event_del (event.c:2188) ==695== by 0x55034A0: consider_writing (bufferevent_openssl.c:881) ==695== by 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119) ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) ==695== by 0x445807: my_client_tunn_event_cb (my_client_tun_intf.c:157) ==695== by 0x52BA162: event_persist_closure (event.c:1301) ==695== by 0x52BA271: event_process_active_single_queue (event.c:1345) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) ==695== by 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650) ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) ==695== by 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== ==695== Invalid write of size 8 ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) ==695== by 0x52BCE7B: event_del_internal (event.c:2251) ==695== by 0x52BCBD7: event_del (event.c:2188) ==695== by 0x550260C: stop_reading (bufferevent_openssl.c:422) ==695== by 0x55028CE: conn_closed (bufferevent_openssl.c:528) ==695== by 0x5502F56: do_write (bufferevent_openssl.c:694) ==695== by 0x55033ED: consider_writing (bufferevent_openssl.c:874) ==695== by 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119) ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) ==695== by 0x445807: my_client_tunn_event_cb (my_client_tun_intf.c:157) ==695== by 0x52BA162: event_persist_closure (event.c:1301) ==695== by 0x52BA271: event_process_active_single_queue (event.c:1345) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) ==695== by 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650) ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) ==695== by 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== ==695== Invalid write of size 8 ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) ==695== by 0x52BCE7B: event_del_internal (event.c:2251) ==695== by 0x52BCBD7: event_del (event.c:2188) ==695== by 0x5502659: stop_writing (bufferevent_openssl.c:435) ==695== by 0x55028DA: conn_closed (bufferevent_openssl.c:529) ==695== by 0x5502F56: do_write (bufferevent_openssl.c:694) ==695== by 0x55033ED: consider_writing (bufferevent_openssl.c:874) ==695== by 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119) ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) ==695== by 0x445807: my_client_tunn_event_cb (my_client_tun_intf.c:157) ==695== by 0x52BA162: event_persist_closure (event.c:1301) ==695== by 0x52BA271: event_process_active_single_queue (event.c:1345) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) ==695== by 0x52C878B: _bufferevent_decref_and_unlock (bufferevent.c:650) ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) ==695== by 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== ==695== Invalid write of size 8 ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) ==695== by 0x52BCE7B: event_del_internal (event.c:2251) ==695== by 0x52BCBD7: event_del (event.c:2188) ==695== by 0x5503FB5: be_openssl_destruct (bufferevent_openssl.c:1183) ==695== by 0x52C85F5: _bufferevent_decref_and_unlock (bufferevent.c:622) ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) ==695== by 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 232 bytes inside a block of size 560 free'd ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) ==695== by 0x41EE2F: my_client_pktbuf_free (my_client_msg.c:54) ==695== by 0x52BFFEC: evbuffer_chain_free (buffer.c:198) ==695== by 0x52C251D: evbuffer_drain (buffer.c:972) ==695== by 0x5502FA2: do_write (bufferevent_openssl.c:703) ==695== by 0x55033ED: consider_writing (bufferevent_openssl.c:874) ==695== by 0x5503D41: be_openssl_outbuf_cb (bufferevent_openssl.c:1119) ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) ==695== by 0x426213: my_client_generic_send_message (my_client_msg.c:1708) ==695== by 0x4297B9: my_client_peer_send_hello (my_client_msg.c:2181) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== ==695== Invalid write of size 8 ==695== at 0x52CF66B: evmap_io_add (evmap.c:328) ==695== by 0x52BC638: event_add_internal (event.c:2073) ==695== by 0x52BC150: event_add (event.c:1966) ==695== by 0x52E695D: _evdns_nameserver_add_impl (evdns.c:2496) ==695== by 0x52E6D7C: evdns_base_nameserver_ip_add (evdns.c:2592) ==695== by 0x52E9B9B: resolv_conf_parse_line (evdns.c:3471) ==695== by 0x52E9ECE: evdns_base_resolv_conf_parse_impl (evdns.c:3572) ==695== by 0x52E9D60: evdns_base_resolv_conf_parse (evdns.c:3508) ==695== by 0x52EA1A7: evdns_base_new (evdns.c:3881) ==695== by 0x41AF33: my_client_connect_to_peer_my_clients (my_client.c:4978) ==695== by 0x430FC6: my_client_peer_timer_exp_cb (my_client_peer.c:285) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 1,480 bytes inside an unallocated block of size 2,960 in arena "client" ==695== ==695== Invalid write of size 8 ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) ==695== by 0x52BCE7B: event_del_internal (event.c:2251) ==695== by 0x52BCBD7: event_del (event.c:2188) ==695== by 0x52EA318: evdns_nameserver_free (evdns.c:3930) ==695== by 0x52EA4EB: evdns_base_free_and_unlock (evdns.c:3965) ==695== by 0x52EA6DD: evdns_base_free (evdns.c:4001) ==695== by 0x41A5AE: my_client_dns_getaddrinfo_cb (my_client.c:4778) ==695== by 0x52EB9E9: evdns_getaddrinfo (evdns.c:4563) ==695== by 0x41B02E: my_client_connect_to_peer_my_clients (my_client.c:5000) ==695== by 0x430FC6: my_client_peer_timer_exp_cb (my_client_peer.c:285) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 1,480 bytes inside an unallocated block of size 2,960 in arena "client" ==695== ==695== Invalid write of size 8 ==695== at 0x52CF66B: evmap_io_add (evmap.c:328) ==695== by 0x52BC638: event_add_internal (event.c:2073) ==695== by 0x52BC150: event_add (event.c:1966) ==695== by 0x431580: ssl_connect_timer_cb (my_client_peer.c:371) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 1,480 bytes inside an unallocated block of size 2,960 in arena "client" ==695== ==695== Invalid write of size 8 ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) ==695== by 0x52BCE7B: event_del_internal (event.c:2251) ==695== by 0x52BCBD7: event_del (event.c:2188) ==695== by 0x52BB484: event_free (event.c:1809) ==695== by 0x435B36: my_client_peer_delete (my_client_peer.c:1323) ==695== by 0x430EE8: my_client_peer_timer_exp_cb (my_client_peer.c:267) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== by 0x41E5BA: main (my_client.c:6024) ==695== Address 0x86ff848 is 8 bytes inside a block of size 160 free'd ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) ==695== by 0x5BFD5F9: CRYPTO_free (mem.c:401) ==695== by 0x5CA9915: EVP_MD_CTX_cleanup (digest.c:390) ==695== by 0x5CA532E: ssleay_rand_bytes (md_rand.c:525) ==695== by 0x5CA53DB: ssleay_rand_pseudo_bytes (md_rand.c:548) ==695== by 0x5CA5C67: RAND_pseudo_bytes (rand_lib.c:173) ==695== by 0x594394A: ssl_fill_hello_random (s23_clnt.c:294) ==695== by 0x5951112: dtls1_client_hello (d1_clnt.c:801) ==695== by 0x59505CB: dtls1_connect (d1_clnt.c:302) ==695== by 0x595C8E0: SSL_connect (ssl_lib.c:943) ==695== by 0x441DBC: create_ssl_conn_to_peer (my_client_misc.c:2134) ==695== by 0x4346AB: my_client_peer_create (my_client_peer.c:1018) ==695== by 0x41B1BB: my_client_connect_to_peer_my_clients (my_client.c:5022) ==695== by 0x41ABF9: my_client_dns_getaddrinfo_cb (my_client.c:4886) ==695== by 0x52EB9E9: evdns_getaddrinfo (evdns.c:4563) ==695== by 0x41B02E: my_client_connect_to_peer_my_clients (my_client.c:5000) ==695== by 0x430FC6: my_client_peer_timer_exp_cb (my_client_peer.c:285) ==695== by 0x488AC6: timer_exec_pri (timer.c:612) ==695== by 0x4885A1: timer_exec (timer.c:504) ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) ==695== by 0x52BA540: event_process_active (event.c:1420) ==695== by 0x52BABA7: event_base_loop (event.c:1621) ==695== by 0x41C5D9: my_client_main (my_client.c:5437) ==695== valgrind: /jenkins/master/builder/x86_64/tmp/work/x86_64-poky-linux/valgrind/3.10.0-r15/valgrind-3.10.0/coregrind/m_mallocfree.c:304 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed. valgrind: Heap block lo/hi size mismatch: lo = 224, hi = 0.
