Hey Björn, multithreading and libevent is a bit tricky (at least it was for me). I would also suggest to change your design... I developed a multithreaded websocket server based on libevent a year ago:
My approach was to create a single event base per thread. Each event base in each thread listens to the same port. Each event base processes its established websocket-connections. For inter-thread-communication i used socketpairs between the event bases in the threads. With this design i was able to achieve ~350.000 connections and a throughput of about 90.000 websocket-messages per second with a ~85-90% CPU load on a quadcore machine with 4 gb ram. Filipe > The posted code compiles on my machine without modifications. Thus I > wonder why you have to do some cleanup. > But the example code fails silently. Perhaps you don't run it with > enough rights to bind to port 443? > > Most of the time I access the server with a browser and keep the > reload button pressed. Sometimes I use the apache ab tool. > > I tried to understand how libevhtp works. If I get it right, all > connections are handled in the main thread and only the payload is > processed in a worker thread. For non-ssl connections this is fine but > for example for a ssl-only static web page server I think I do not > really profit from multithreading this way because all of the ssl > stuff is done in the main thread. > > 2013/1/8 Nir Soffer <nir...@gmail.com>: >> >> On Jan 8, 2013, at 12:55 AM, Björn K. wrote: >> >> That piece of code should distribute the incoming connections over the >> threads (I want to change that code later to select the thread with >> the lowest number of handled connections). If I don't use this code >> and use instead >> bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx, >> ... >> the problem remains. >> >> Furthermore I don't think it's a problem of running out of memory. The >> segmentation fault even occurs when there are just a few connections >> to the server (once the third connection produced the crash). >> I tried to get a segmentation fault during a run in valgrind. One time >> I succeeded and got this: >> >> ==310== Invalid read of size 8 >> ==310== at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626) >> ==310== by 0x41403F: bufferevent_enable (bufferevent.c:448) >> ==310== by 0x4044A9: acceptcb (in /home/login/test/simple) >> ==310== by 0x424325: listener_read_cb (listener.c:412) >> ==310== by 0x41D889: event_process_active_single_queue (event.c:1471) >> ==310== by 0x41E0E6: event_base_loop (event.c:1538) >> ==310== by 0x4049E0: main (in /home/login/test/simple) >> ==310== Address 0x62531a0 is 464 bytes inside a block of size 560 >> free'd >> ==310== at 0x4C240FD: free (vg_replace_malloc.c:366) >> ==310== by 0x413E2A: bufferevent_decref_and_unlock_ >> (bufferevent.c:703) >> ==310== by 0x41DAD4: event_process_active_single_queue (event.c:1476) >> ==310== by 0x41E0E6: event_base_loop (event.c:1538) >> ==310== by 0x404693: worker_function (in /home/login/test/simple) >> ==310== by 0x54258C9: start_thread (pthread_create.c:300) >> ==310== >> >> (And many similar messages. What they have in common is that some >> memory is freed in the worker thread and the main thread tries to >> access it.) >> >> But I don't know why memory of the bufferevent can be freed in the >> worker thread before the buffer event is enabled. >> >> >> I think the key to solve these issues is change the design - do not use >> libevent apis from multiple threads except event_active(). >> >> The simplest solution would be to use libevhtp >> <https://github.com/ellzey/libevhtp/> which uses smart event based >> thread >> pool and looks very clean. >> >> If you want more control, the simplest solution is to get a single >> process >> working, then fork few worker processes in the main process, all sharing >> the >> same listener file descriptor, bound in the main process. Then let the >> kernel do the load balancing for you. Successful projects like nginx and >> unicorn <https://github.com/blog/517-unicorn> work like this. >> >> When I try the code you submitted (after some cleanup to make it compile >> and >> run), it does not respond when accessing with a browser. How are you >> stress >> testing this code? >> >> >> 2013/1/7 Mark Ellzey <mtho...@strcpy.net>: >> >> On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote: >> >> >> >> at a second glance: >> >> >> next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is >> >> working as intended? >> >> *********************************************************************** >> >> To unsubscribe, send an e-mail to majord...@freehaven.net with >> >> unsubscribe libevent-users in the body. >> >> *********************************************************************** >> To unsubscribe, send an e-mail to majord...@freehaven.net with >> unsubscribe libevent-users in the body. >> >> > *********************************************************************** > To unsubscribe, send an e-mail to majord...@freehaven.net with > unsubscribe libevent-users in the body. > *********************************************************************** To unsubscribe, send an e-mail to majord...@freehaven.net with unsubscribe libevent-users in the body.
