Hi Azat We upgraded our libevent to the latest code base https://github.com/libevent/libevent, We DO NOT see the crash anymore (Yay!!). This was very easily reproducible crash under stress, looks like you folks have fixed this in your latest code base.
Can you please point us to the exact patch which fixes this crash ? Or do you know if your next STABLE release will have a fix for this ? Please throw some light on as to how to proceed. Thanks in Advance -Praveen On Tue, Feb 3, 2015 at 2:23 PM, Azat Khuzhin <[email protected]> wrote: > On Tue, Feb 03, 2015 at 01:39:22PM -0800, Praveen Kariyanahalli wrote: > > Client (bufferevent + openssl) crash *only under load situation*. > Valgrind > > points to this backtrace. Any help is greatly appreciated ? Please let us > > know if you need any more information. > > > > Thanks > > -Praveen > > > > Version: 2.0.22 > > Hi Praveen, > > Could you try more recent version of libevent? > Preferably compile from sources using the latest code base at: > https://github.com/libevent/libevent > > > > > ==695== Invalid write of size 8 > > ==695== at 0x52CF66B: evmap_io_add (evmap.c:328) > > ==695== by 0x52BC638: event_add_internal (event.c:2073) > > ==695== by 0x52BC150: event_add (event.c:1966) > > ==695== by 0x5504678: bufferevent_openssl_new_impl > > (bufferevent_openssl.c:1368) > > ==695== by 0x550482B: bufferevent_openssl_socket_new > > (bufferevent_openssl.c:1440) > > ==695== by 0x43211A: tls_connect_timer_cb (my_client_peer.c:528) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd > > ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) > > ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) > > ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) > > ==695== by 0x52C878B: _bufferevent_decref_and_unlock > (bufferevent.c:650) > > Your program multithreaded? > > Could you strip you program to ~50-100L source, that will reproduce this > problem? > > Cheers, > Azat. > > > ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) > > ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) > > ==695== by 0x430EE8: my_client_peer_timer_exp_cb > (my_client_peer.c:267) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) > > ==695== by 0x52BCE7B: event_del_internal (event.c:2251) > > ==695== by 0x52BCBD7: event_del (event.c:2188) > > ==695== by 0x55034A0: consider_writing (bufferevent_openssl.c:881) > > ==695== by 0x5503D41: be_openssl_outbuf_cb > (bufferevent_openssl.c:1119) > > ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) > > ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) > > ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) > > ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) > > ==695== by 0x445807: my_client_tunn_event_cb > (my_client_tun_intf.c:157) > > ==695== by 0x52BA162: event_persist_closure (event.c:1301) > > ==695== by 0x52BA271: event_process_active_single_queue (event.c:1345) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd > > ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) > > ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) > > ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) > > ==695== by 0x52C878B: _bufferevent_decref_and_unlock > (bufferevent.c:650) > > ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) > > ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) > > ==695== by 0x430EE8: my_client_peer_timer_exp_cb > (my_client_peer.c:267) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) > > ==695== by 0x52BCE7B: event_del_internal (event.c:2251) > > ==695== by 0x52BCBD7: event_del (event.c:2188) > > ==695== by 0x550260C: stop_reading (bufferevent_openssl.c:422) > > ==695== by 0x55028CE: conn_closed (bufferevent_openssl.c:528) > > ==695== by 0x5502F56: do_write (bufferevent_openssl.c:694) > > ==695== by 0x55033ED: consider_writing (bufferevent_openssl.c:874) > > ==695== by 0x5503D41: be_openssl_outbuf_cb > (bufferevent_openssl.c:1119) > > ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) > > ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) > > ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) > > ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) > > ==695== by 0x445807: my_client_tunn_event_cb > (my_client_tun_intf.c:157) > > ==695== by 0x52BA162: event_persist_closure (event.c:1301) > > ==695== by 0x52BA271: event_process_active_single_queue (event.c:1345) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd > > ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) > > ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) > > ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) > > ==695== by 0x52C878B: _bufferevent_decref_and_unlock > (bufferevent.c:650) > > ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) > > ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) > > ==695== by 0x430EE8: my_client_peer_timer_exp_cb > (my_client_peer.c:267) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) > > ==695== by 0x52BCE7B: event_del_internal (event.c:2251) > > ==695== by 0x52BCBD7: event_del (event.c:2188) > > ==695== by 0x5502659: stop_writing (bufferevent_openssl.c:435) > > ==695== by 0x55028DA: conn_closed (bufferevent_openssl.c:529) > > ==695== by 0x5502F56: do_write (bufferevent_openssl.c:694) > > ==695== by 0x55033ED: consider_writing (bufferevent_openssl.c:874) > > ==695== by 0x5503D41: be_openssl_outbuf_cb > (bufferevent_openssl.c:1119) > > ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) > > ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) > > ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) > > ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) > > ==695== by 0x445807: my_client_tunn_event_cb > (my_client_tun_intf.c:157) > > ==695== by 0x52BA162: event_persist_closure (event.c:1301) > > ==695== by 0x52BA271: event_process_active_single_queue (event.c:1345) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 232 bytes inside a block of size 608 free'd > > ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) > > ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) > > ==695== by 0x52BE3F7: event_mm_free_ (event.c:2696) > > ==695== by 0x52C878B: _bufferevent_decref_and_unlock > (bufferevent.c:650) > > ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) > > ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) > > ==695== by 0x430EE8: my_client_peer_timer_exp_cb > (my_client_peer.c:267) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) > > ==695== by 0x52BCE7B: event_del_internal (event.c:2251) > > ==695== by 0x52BCBD7: event_del (event.c:2188) > > ==695== by 0x5503FB5: be_openssl_destruct (bufferevent_openssl.c:1183) > > ==695== by 0x52C85F5: _bufferevent_decref_and_unlock > (bufferevent.c:622) > > ==695== by 0x52C8874: bufferevent_free (bufferevent.c:681) > > ==695== by 0x435747: my_client_peer_delete (my_client_peer.c:1257) > > ==695== by 0x430EE8: my_client_peer_timer_exp_cb > (my_client_peer.c:267) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 232 bytes inside a block of size 560 free'd > > ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) > > ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) > > ==695== by 0x41EE2F: my_client_pktbuf_free (my_client_msg.c:54) > > ==695== by 0x52BFFEC: evbuffer_chain_free (buffer.c:198) > > ==695== by 0x52C251D: evbuffer_drain (buffer.c:972) > > ==695== by 0x5502FA2: do_write (bufferevent_openssl.c:703) > > ==695== by 0x55033ED: consider_writing (bufferevent_openssl.c:874) > > ==695== by 0x5503D41: be_openssl_outbuf_cb > (bufferevent_openssl.c:1119) > > ==695== by 0x52C0B58: evbuffer_run_callbacks (buffer.c:486) > > ==695== by 0x52C0C51: evbuffer_invoke_callbacks (buffer.c:508) > > ==695== by 0x52C61AD: evbuffer_add_reference (buffer.c:2718) > > ==695== by 0x41EF6E: my_client_pkt_send (my_client_msg.c:75) > > ==695== by 0x426213: my_client_generic_send_message > > (my_client_msg.c:1708) > > ==695== by 0x4297B9: my_client_peer_send_hello (my_client_msg.c:2181) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF66B: evmap_io_add (evmap.c:328) > > ==695== by 0x52BC638: event_add_internal (event.c:2073) > > ==695== by 0x52BC150: event_add (event.c:1966) > > ==695== by 0x52E695D: _evdns_nameserver_add_impl (evdns.c:2496) > > ==695== by 0x52E6D7C: evdns_base_nameserver_ip_add (evdns.c:2592) > > ==695== by 0x52E9B9B: resolv_conf_parse_line (evdns.c:3471) > > ==695== by 0x52E9ECE: evdns_base_resolv_conf_parse_impl (evdns.c:3572) > > ==695== by 0x52E9D60: evdns_base_resolv_conf_parse (evdns.c:3508) > > ==695== by 0x52EA1A7: evdns_base_new (evdns.c:3881) > > ==695== by 0x41AF33: my_client_connect_to_peer_my_clients > > (my_client.c:4978) > > ==695== by 0x430FC6: my_client_peer_timer_exp_cb > (my_client_peer.c:285) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 1,480 bytes inside an unallocated block of > > size 2,960 in arena "client" > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) > > ==695== by 0x52BCE7B: event_del_internal (event.c:2251) > > ==695== by 0x52BCBD7: event_del (event.c:2188) > > ==695== by 0x52EA318: evdns_nameserver_free (evdns.c:3930) > > ==695== by 0x52EA4EB: evdns_base_free_and_unlock (evdns.c:3965) > > ==695== by 0x52EA6DD: evdns_base_free (evdns.c:4001) > > ==695== by 0x41A5AE: my_client_dns_getaddrinfo_cb (my_client.c:4778) > > ==695== by 0x52EB9E9: evdns_getaddrinfo (evdns.c:4563) > > ==695== by 0x41B02E: my_client_connect_to_peer_my_clients > > (my_client.c:5000) > > ==695== by 0x430FC6: my_client_peer_timer_exp_cb > (my_client_peer.c:285) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 1,480 bytes inside an unallocated block of > > size 2,960 in arena "client" > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF66B: evmap_io_add (evmap.c:328) > > ==695== by 0x52BC638: event_add_internal (event.c:2073) > > ==695== by 0x52BC150: event_add (event.c:1966) > > ==695== by 0x431580: ssl_connect_timer_cb (my_client_peer.c:371) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 1,480 bytes inside an unallocated block of > > size 2,960 in arena "client" > > ==695== > > ==695== Invalid write of size 8 > > ==695== at 0x52CF8DD: evmap_io_del (evmap.c:384) > > ==695== by 0x52BCE7B: event_del_internal (event.c:2251) > > ==695== by 0x52BCBD7: event_del (event.c:2188) > > ==695== by 0x52BB484: event_free (event.c:1809) > > ==695== by 0x435B36: my_client_peer_delete (my_client_peer.c:1323) > > ==695== by 0x430EE8: my_client_peer_timer_exp_cb > (my_client_peer.c:267) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== by 0x41E5BA: main (my_client.c:6024) > > ==695== Address 0x86ff848 is 8 bytes inside a block of size 160 free'd > > ==695== at 0x4A0739B: free (vg_replace_malloc.c:473) > > ==695== by 0x48B3EB: vip_guard_mem_free (vip_gaurd_mem.c:157) > > ==695== by 0x5BFD5F9: CRYPTO_free (mem.c:401) > > ==695== by 0x5CA9915: EVP_MD_CTX_cleanup (digest.c:390) > > ==695== by 0x5CA532E: ssleay_rand_bytes (md_rand.c:525) > > ==695== by 0x5CA53DB: ssleay_rand_pseudo_bytes (md_rand.c:548) > > ==695== by 0x5CA5C67: RAND_pseudo_bytes (rand_lib.c:173) > > ==695== by 0x594394A: ssl_fill_hello_random (s23_clnt.c:294) > > ==695== by 0x5951112: dtls1_client_hello (d1_clnt.c:801) > > ==695== by 0x59505CB: dtls1_connect (d1_clnt.c:302) > > ==695== by 0x595C8E0: SSL_connect (ssl_lib.c:943) > > ==695== by 0x441DBC: create_ssl_conn_to_peer (my_client_misc.c:2134) > > ==695== by 0x4346AB: my_client_peer_create (my_client_peer.c:1018) > > ==695== by 0x41B1BB: my_client_connect_to_peer_my_clients > > (my_client.c:5022) > > ==695== by 0x41ABF9: my_client_dns_getaddrinfo_cb (my_client.c:4886) > > ==695== by 0x52EB9E9: evdns_getaddrinfo (evdns.c:4563) > > ==695== by 0x41B02E: my_client_connect_to_peer_my_clients > > (my_client.c:5000) > > ==695== by 0x430FC6: my_client_peer_timer_exp_cb > (my_client_peer.c:285) > > ==695== by 0x488AC6: timer_exec_pri (timer.c:612) > > ==695== by 0x4885A1: timer_exec (timer.c:504) > > ==695== by 0x41A165: my_client_base_timer_cb (my_client.c:4671) > > ==695== by 0x52BA2CC: event_process_active_single_queue (event.c:1350) > > ==695== by 0x52BA540: event_process_active (event.c:1420) > > ==695== by 0x52BABA7: event_base_loop (event.c:1621) > > ==695== by 0x41C5D9: my_client_main (my_client.c:5437) > > ==695== > > > > valgrind: > > > /jenkins/master/builder/x86_64/tmp/work/x86_64-poky-linux/valgrind/3.10.0-r15/valgrind-3.10.0/coregrind/m_mallocfree.c:304 > > (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed. > > valgrind: Heap block lo/hi size mismatch: lo = 224, hi = 0. > *********************************************************************** > To unsubscribe, send an e-mail to [email protected] with > unsubscribe libevent-users in the body. > -- Regards -Praveen
