On Sat, Jan 03, 2015 at 08:22:52PM +0000, Gerry Sweeney wrote: > Hi Azat, > > Ok thanks. You are right, there is not much different in the two patterns, > they are almost the same apart from the first pattern the request is served > as soon as there is a free thread to serve it and the second is essentially > an open connection that can have data sent down it at any time by another > thread. >
first pattern -> libevhtp As for the second pattern, am I understand correct, that the main goal of it is is nhave separate thread for non-idle connection? IOW we can't reuse that separate thread to do some job for other connections. If so, I don't think that this is AIO's task, *but* all you need is just move handling of this request into separate thread, and a brief look at libevhtp shows me that it is pretty easy to add. > If libevent already works like this how do I control how many worker threads > it will use? Are they created on demand? or in a pool? This is the sort of > information thats really unclear to me. I have have a look in the source code > as you suggest to see if I can get a better understanding. > This is not libevent, but libevhtp: https://github.com/ellzey/libevhtp/blob/master/examples/thread_design.c In short: evhtp_use_threads() > Gerry > > > > Gerry Sweeney > http://gerrysweeney.com/ > > > > On 3 Jan 2015, at 17:43, Azat Khuzhin <[email protected]> wrote: > > > > On Sat, Jan 03, 2015 at 08:41:31PM +0300, Azat Khuzhin wrote: > >> On Sat, Jan 03, 2015 at 05:02:36PM +0000, Gerry Sweeney wrote: > >>> Hi Azat, > >>> > >>> Thank you for the detailed response and sorry for the late reply, I do > >>> really appreciate your time to respond - thank you. May I also appologize > >>> for the incompleteness of the example, I lifted the code out of a bigger > >>> project and failed to compile it stand-alone before I sent it - lesson > >>> learned :) > >>> > >>> Yes that fixed the problem, it also highlighted another problem, I was > >>> linking against an earlier version of the lib event library so the > >>> evthread_use_pthreads() could not be resolved at link time, I have also > >>> corrected that problem and also used “event2/event.h” instead of > >>> “event.h”. It is not that obvious from the various examples that there > >>> are different headers and library versions to contend with (thats my > >>> excuse anyway :) > >> > >> event.h is just a deprecated wrapper for event2/*, so it must works with > >> event.h too (But *yes, it is deprecated, so please don't use it*). > >> > >>> > >>> It now works as expected so thank you for your help. I did try to use > >>> libevhtp but I could not get it to compile, I think this was most likely > >>> down to the fact I was including version 1x of the lib event, I will try > >>> it again. > >> > >> AFAIR libevhtp needs libevent 2. > >> > >>> > >>> I am trying to create two different HTTP server patterns: - > >>> > >>> First one is a simple multi-threaded web server, where I receive requests > >>> asynchronously complete HTTP requests, event driven into a buffer (so > >>> presumably a buffered event) - this is the async part. Once I receive a > >>> valid HTTP request from the client, I put the request (and its > >>> connection) into a queue to be serviced synchronously by a thread from a > >>> pool of worker threads, ones the request has been served, the connection > >>> is once again put into the async receive queue ready for the next > >>> request. > >>> > >>> The second pattern, I want to create a server to be able to accept a > >>> large number of long-term and generally idle connections. Like the first > >>> example, the event driven receive will handle the request part, and once > >>> a valid request has been received the connection will be added to a > >>> connection list. At any point, other threads in the server will be able > >>> to send data to any clients open connection by looking up the open > >>> connection in the connection list. To visualise in HTTP terms the > >>> response data would most likely look like a never-ending chunked > >>> response, each chunk would be sent as times dictated by the server system. > >> > >> I don't see *big* difference between this two patterns (but maybe I > >> missed something?): > >> 1) accept() -> do job in a thread -> wait for other requests from client > >> 2) accept() -> do job in a thread (send chunked responses forever) > >> > >> And libevhtp works just like this, it accept() in the main thread, and > >> after this schedule callbacks into separate thread pool, by using pipes > >> (pipe(2)). > > > > Sorry, not pipes, but sockets (i.e. socketpair(2)) > > > >> > >> Here is brief explanation how this part works in libevhtp: > >> - evhtp.c: > >> _evhtp_accept_cb() -> evthr_pool_defer() > >> - evthr.c: > >> evthr_pool_defer() -> evthr_defer() -> send() -> ... -> _evthr_read_cmd() > >> > >> And _evthr_read_cmd() works in a separate thread, while send() in main. > >> > >> And please could you avoid top-posting? > >> > >> Cheers, > >> Azat. > >> > >>> > >>> I am sure there are more formal names/descriptions for these patterns but > >>> thats as I understand them. These patterns are both terribly important > >>> for web-scale application design so they are worthy of some good > >>> documentation I think. > >>> > >>> For me personally, I thought it would be good to learn how to do this > >>> using libevent just as an exercise to learn how the library and event > >>> model implemented by libevent works, but so far its not been terribly > >>> obvious how I should approach either of these. I tried the first pattern > >>> but failed to make it work, evthread_use_pthreads() might solve this now > >>> so I will need to test. > >>> > >>> It would be terribly helpful if someone could point me in the right > >>> general direction with some pseudo code, which functions I would call to > >>> achieve those patterns, I know that may be asking a bit much but I > >>> thought I would ask anyway. > >>> > >>> For what its worth, I will make an effort do document my efforts on these > >>> two patterns on my blog http://gerrysweeney.com/ in order to hopefully > >>> feedback to the community. > >>> > >>> Gerry > >>> > >>>> > >>> > >>> > >>> Gerry Sweeney > >>> http://gerrysweeney.com/ > >>> > >>> > >>>> On 30 Dec 2014, at 16:00, Azat Khuzhin <[email protected]> wrote: > >>>> > >>>> On Sun, Dec 28, 2014 at 06:49:28PM +0000, Gerry Sweeney wrote: > >>>>> Hi Azat, > >>>>> > >>>>> See below, this is the program, see the comments in main() for an > >>>>> explanation of the problem. This needs to be a c++ compile as I am > >>>>> using an STL <vector>. I have made it work by freeing the event_base > >>>>> object by calling free_event_base() but that feels like the wrong thing > >>>>> to do. > >>>>> > >>>>> Thanks for the response, I appreciate it…any help would be much > >>>>> appreciated > >>>> > >>>> Please see comments for code, in short all you need in > >>>> evthread_use_pthreads(), but you code have some design issues and also > >>>> errors that don't allow me to compile this simple program. > >>>> > >>>>> > >>>>> Gerry > >>>>> > >>>>> ---------------------------------- > >>>>> #include <sys/types.h> > >>>>> #include <sys/time.h> > >>>>> #include <sys/queue.h> > >>>>> #include <stdlib.h> > >>>>> #include <unistd.h> > >>>>> #include <err.h> > >>>>> #include <event.h> > >>>>> #include <evhttp.h> > >>>>> #include <fcntl.h> > >>>>> #include <sys/socket.h> > >>>>> #include <netinet/in.h> > >>>>> #include <iostream> > >>>>> #include <vector> > >>>>> > >>>>> void httpserver_ProcessRequest(struct evhttp_request *req) { > >>>>> struct evbuffer *buf = evbuffer_new(); > >>>>> if (buf == NULL) > >>>>> return; > >>>>> > >>>>> evbuffer_add_printf(buf, "Requested: %s", evhttp_request_uri(req)); > >>>>> > >>>>> evhttp_send_reply(req, HTTP_OK, "OK", buf); > >>>>> > >>>>> evbuffer_free(buf); > >>>>> } > >>>>> > >>>>> void* httpserver_Dispatch(void *arg) { > >>>>> > >>>>> event_base_dispatch((struct event_base*)arg); > >>>>> > >>>>> std::cout << "Event dispatch thread ended..." << std::endl; > >>>>> > >>>>> return 0; > >>>>> } > >>>>> > >>>>> int httpserver_bindsocket(int port, int backlog) > >>>>> { > >>>>> int r; > >>>>> int nfd; > >>>>> nfd = socket(AF_INET, SOCK_STREAM, 0); > >>>>> if (nfd < 0) return -1; > >>>>> > >>>>> int one = 1; > >>>>> r = setsockopt(nfd, SOL_SOCKET, SO_REUSEADDR, (char *)&one, > >>>>> sizeof(int)); > >>>>> > >>>>> struct sockaddr_in addr; > >>>>> memset(&addr, 0, sizeof(addr)); > >>>>> addr.sin_family = AF_INET; > >>>>> addr.sin_addr.s_addr = INADDR_ANY; > >>>>> addr.sin_port = htons(port); > >>>>> > >>>>> r = bind(nfd, (struct sockaddr*)&addr, sizeof(addr)); > >>>>> if (r < 0) return -1; > >>>>> r = listen(nfd, backlog); > >>>>> if (r < 0) return -1; > >>>>> > >>>>> int flags; > >>>>> if ((flags = fcntl(nfd, F_GETFL, 0)) < 0 > >>>>> || fcntl(nfd, F_SETFL, flags | O_NONBLOCK) < 0) > >>>>> return -1; > >>>>> > >>>>> return nfd; > >>>>> } > >>>>> > >>>>> int httpserver_start(int port, int nthreads, int backlog) > >>>>> { > >>>>> int r, i; > >>>>> int nfd = httpserver_bindsocket(port, backlog); > >>>>> if (nfd < 0) > >>>>> return -1; > >>>>> > >>>>> struct workder_t > >>>>> { > >>>>> struct event_base* base; > >>>>> struct evhttp* httpd; > >>>>> struct evhttp_bound_socket* soc; > >>>>> pthread_t thr; > >>>>> }; > >>>>> > >>>>> std::vector<workder_t> _workers; > >>>>> > >>>>> for (i = 0; i < nthreads; i++) > >>>>> { > >>>>> workder_t worker; > >>>>> > >>>>> worker.base = event_base_new(); > >>>>> if (worker.base == NULL) > >>>>> return -1; > >>>>> > >>>>> worker.httpd = evhttp_new(worker.base); > >>>>> if (worker.httpd == NULL) > >>>>> return -1; > >>>>> > >>>>> worker.soc = evhttp_accept_socket_with_handle(worker.httpd, nfd); > >>>>> > >>>>> evhttp_set_gencb(worker.httpd, httpserver_GenericHandler, NULL); > >>>> > >>>> No such function -- httpserver_GenericHandler. > >>>> > >>>>> > >>>>> r = pthread_create(&worker.thr, NULL, httpserver_Dispatch, > >>>>> worker.base); > >>>>> > >>>>> if (r != 0) > >>>>> return -1; > >>>>> > >>>>> _workers.push_back(worker); > >>>>> } > >>>>> > >>>>> std::cout << "Running - press return to stop..." << std::endl; > >>>>> getchar(); > >>>>> std::cout << "Stopping..." << std::endl; > >>>>> > >>>>> close(nfd); > >>>>> std::cout << "Closed listening socket..." << std::endl; > >>>>> > >>>>> for(auto worker = _workers.begin(); worker != _workers.end(); > >>>>> ++worker) > >>>>> { > >>>>> // I would expect that this would cause the thread blocked in > >>>>> event_base_dispatch() function called in httpserver_Dispatch function > >>>>> to end > >>>>> event_base_loopbreak(worker->base); > >>>>> > >>>>> // seems like I should do this, but not sure…works either > >>>>> way > >>>>> evhttp_free(worker->httpd); > >>>> > >>>> This will close accept socket, because of LEV_OPT_CLOSE_ON_FREE in > >>>> evhttp_accept_socket_with_handle(), you must use evhttp_bind_listener() > >>>> here to avoid this. > >>>> > >>>> You could look into libevhtp, it has thread support out-of-the-box, and > >>>> it is simpler since it is library that just implements http server. > >>>> > >>>>> > >>>>> // If I do this the the event_base_dispatch() does return so I > >>>>> get the expected result, but this feels wrong to > >>>>> // destroy the base before the thread ends > >>>>> // comment the next line to see the problem > >>>>> event_base_free(worker->base); > >>>> > >>>> Here you first freeing base, and then in separate thread > >>>> event_base_dispatch() will use some internal fields in it after free -- > >>>> IOW you must free all resources after pthread_join(). > >>>> > >>>>> > >>>>> // We are in the main thread so we join with each worker thread > >>>>> here > >>>>> pthread_join(worker->thr, NULL); > >>>>> } > >>>>> std::cout << “Closed..." << std::endl; > >>>> > >>>> Bad quotes. > >>>> > >>>>> > >>>>> return 0; > >>>>> } > >>>>> > >>>>> int main(int argc, char* argv[]) > >>>>> { > >>>> > >>>> In short all you need is add the next line here: > >>>> evthread_use_pthreads(); > >>>> > >>>> This will enable inter-thread notification for loop break. > >>>> > >>>> [ I do have some other changes to you program, so if just adding > >>>> evthread_use_pthreads() didn't work for you -- let me know, I will send > >>>> you patch with my local changes ]. > >>>> > >>>>> httpserver_start(8080, NUM_THREADS, 100); > >>>> > >>>> NUM_THREADS undefined. > >>>> *********************************************************************** > >>>> To unsubscribe, send an e-mail to [email protected] with > >>>> unsubscribe libevent-users in the body. > >>> > >>> *********************************************************************** > >>> To unsubscribe, send an e-mail to [email protected] with > >>> unsubscribe libevent-users in the body. > >> > >> -- > >> Respectfully > >> Azat Khuzhin > > > > -- > > Respectfully > > Azat Khuzhin > > *********************************************************************** > > To unsubscribe, send an e-mail to [email protected] with > > unsubscribe libevent-users in the body. > > *********************************************************************** > To unsubscribe, send an e-mail to [email protected] with > unsubscribe libevent-users in the body. -- Respectfully Azat Khuzhin *********************************************************************** To unsubscribe, send an e-mail to [email protected] with unsubscribe libevent-users in the body.
