Hi Azat, Ok thanks. You are right, there is not much different in the two patterns, they are almost the same apart from the first pattern the request is served as soon as there is a free thread to serve it and the second is essentially an open connection that can have data sent down it at any time by another thread.
If libevent already works like this how do I control how many worker threads it will use? Are they created on demand? or in a pool? This is the sort of information thats really unclear to me. I have have a look in the source code as you suggest to see if I can get a better understanding. Gerry Gerry Sweeney http://gerrysweeney.com/ > On 3 Jan 2015, at 17:43, Azat Khuzhin <[email protected]> wrote: > > On Sat, Jan 03, 2015 at 08:41:31PM +0300, Azat Khuzhin wrote: >> On Sat, Jan 03, 2015 at 05:02:36PM +0000, Gerry Sweeney wrote: >>> Hi Azat, >>> >>> Thank you for the detailed response and sorry for the late reply, I do >>> really appreciate your time to respond - thank you. May I also appologize >>> for the incompleteness of the example, I lifted the code out of a bigger >>> project and failed to compile it stand-alone before I sent it - lesson >>> learned :) >>> >>> Yes that fixed the problem, it also highlighted another problem, I was >>> linking against an earlier version of the lib event library so the >>> evthread_use_pthreads() could not be resolved at link time, I have also >>> corrected that problem and also used “event2/event.h” instead of “event.h”. >>> It is not that obvious from the various examples that there are different >>> headers and library versions to contend with (thats my excuse anyway :) >> >> event.h is just a deprecated wrapper for event2/*, so it must works with >> event.h too (But *yes, it is deprecated, so please don't use it*). >> >>> >>> It now works as expected so thank you for your help. I did try to use >>> libevhtp but I could not get it to compile, I think this was most likely >>> down to the fact I was including version 1x of the lib event, I will try it >>> again. >> >> AFAIR libevhtp needs libevent 2. >> >>> >>> I am trying to create two different HTTP server patterns: - >>> >>> First one is a simple multi-threaded web server, where I receive requests >>> asynchronously complete HTTP requests, event driven into a buffer (so >>> presumably a buffered event) - this is the async part. Once I receive a >>> valid HTTP request from the client, I put the request (and its connection) >>> into a queue to be serviced synchronously by a thread from a pool of worker >>> threads, ones the request has been served, the connection is once again put >>> into the async receive queue ready for the next request. >>> >>> The second pattern, I want to create a server to be able to accept a large >>> number of long-term and generally idle connections. Like the first example, >>> the event driven receive will handle the request part, and once a valid >>> request has been received the connection will be added to a connection >>> list. At any point, other threads in the server will be able to send data >>> to any clients open connection by looking up the open connection in the >>> connection list. To visualise in HTTP terms the response data would most >>> likely look like a never-ending chunked response, each chunk would be sent >>> as times dictated by the server system. >> >> I don't see *big* difference between this two patterns (but maybe I >> missed something?): >> 1) accept() -> do job in a thread -> wait for other requests from client >> 2) accept() -> do job in a thread (send chunked responses forever) >> >> And libevhtp works just like this, it accept() in the main thread, and >> after this schedule callbacks into separate thread pool, by using pipes >> (pipe(2)). > > Sorry, not pipes, but sockets (i.e. socketpair(2)) > >> >> Here is brief explanation how this part works in libevhtp: >> - evhtp.c: >> _evhtp_accept_cb() -> evthr_pool_defer() >> - evthr.c: >> evthr_pool_defer() -> evthr_defer() -> send() -> ... -> _evthr_read_cmd() >> >> And _evthr_read_cmd() works in a separate thread, while send() in main. >> >> And please could you avoid top-posting? >> >> Cheers, >> Azat. >> >>> >>> I am sure there are more formal names/descriptions for these patterns but >>> thats as I understand them. These patterns are both terribly important for >>> web-scale application design so they are worthy of some good documentation >>> I think. >>> >>> For me personally, I thought it would be good to learn how to do this using >>> libevent just as an exercise to learn how the library and event model >>> implemented by libevent works, but so far its not been terribly obvious how >>> I should approach either of these. I tried the first pattern but failed to >>> make it work, evthread_use_pthreads() might solve this now so I will need >>> to test. >>> >>> It would be terribly helpful if someone could point me in the right general >>> direction with some pseudo code, which functions I would call to achieve >>> those patterns, I know that may be asking a bit much but I thought I would >>> ask anyway. >>> >>> For what its worth, I will make an effort do document my efforts on these >>> two patterns on my blog http://gerrysweeney.com/ in order to hopefully >>> feedback to the community. >>> >>> Gerry >>> >>>> >>> >>> >>> Gerry Sweeney >>> http://gerrysweeney.com/ >>> >>> >>>> On 30 Dec 2014, at 16:00, Azat Khuzhin <[email protected]> wrote: >>>> >>>> On Sun, Dec 28, 2014 at 06:49:28PM +0000, Gerry Sweeney wrote: >>>>> Hi Azat, >>>>> >>>>> See below, this is the program, see the comments in main() for an >>>>> explanation of the problem. This needs to be a c++ compile as I am using >>>>> an STL <vector>. I have made it work by freeing the event_base object by >>>>> calling free_event_base() but that feels like the wrong thing to do. >>>>> >>>>> Thanks for the response, I appreciate it…any help would be much >>>>> appreciated >>>> >>>> Please see comments for code, in short all you need in >>>> evthread_use_pthreads(), but you code have some design issues and also >>>> errors that don't allow me to compile this simple program. >>>> >>>>> >>>>> Gerry >>>>> >>>>> ---------------------------------- >>>>> #include <sys/types.h> >>>>> #include <sys/time.h> >>>>> #include <sys/queue.h> >>>>> #include <stdlib.h> >>>>> #include <unistd.h> >>>>> #include <err.h> >>>>> #include <event.h> >>>>> #include <evhttp.h> >>>>> #include <fcntl.h> >>>>> #include <sys/socket.h> >>>>> #include <netinet/in.h> >>>>> #include <iostream> >>>>> #include <vector> >>>>> >>>>> void httpserver_ProcessRequest(struct evhttp_request *req) { >>>>> struct evbuffer *buf = evbuffer_new(); >>>>> if (buf == NULL) >>>>> return; >>>>> >>>>> evbuffer_add_printf(buf, "Requested: %s", evhttp_request_uri(req)); >>>>> >>>>> evhttp_send_reply(req, HTTP_OK, "OK", buf); >>>>> >>>>> evbuffer_free(buf); >>>>> } >>>>> >>>>> void* httpserver_Dispatch(void *arg) { >>>>> >>>>> event_base_dispatch((struct event_base*)arg); >>>>> >>>>> std::cout << "Event dispatch thread ended..." << std::endl; >>>>> >>>>> return 0; >>>>> } >>>>> >>>>> int httpserver_bindsocket(int port, int backlog) >>>>> { >>>>> int r; >>>>> int nfd; >>>>> nfd = socket(AF_INET, SOCK_STREAM, 0); >>>>> if (nfd < 0) return -1; >>>>> >>>>> int one = 1; >>>>> r = setsockopt(nfd, SOL_SOCKET, SO_REUSEADDR, (char *)&one, >>>>> sizeof(int)); >>>>> >>>>> struct sockaddr_in addr; >>>>> memset(&addr, 0, sizeof(addr)); >>>>> addr.sin_family = AF_INET; >>>>> addr.sin_addr.s_addr = INADDR_ANY; >>>>> addr.sin_port = htons(port); >>>>> >>>>> r = bind(nfd, (struct sockaddr*)&addr, sizeof(addr)); >>>>> if (r < 0) return -1; >>>>> r = listen(nfd, backlog); >>>>> if (r < 0) return -1; >>>>> >>>>> int flags; >>>>> if ((flags = fcntl(nfd, F_GETFL, 0)) < 0 >>>>> || fcntl(nfd, F_SETFL, flags | O_NONBLOCK) < 0) >>>>> return -1; >>>>> >>>>> return nfd; >>>>> } >>>>> >>>>> int httpserver_start(int port, int nthreads, int backlog) >>>>> { >>>>> int r, i; >>>>> int nfd = httpserver_bindsocket(port, backlog); >>>>> if (nfd < 0) >>>>> return -1; >>>>> >>>>> struct workder_t >>>>> { >>>>> struct event_base* base; >>>>> struct evhttp* httpd; >>>>> struct evhttp_bound_socket* soc; >>>>> pthread_t thr; >>>>> }; >>>>> >>>>> std::vector<workder_t> _workers; >>>>> >>>>> for (i = 0; i < nthreads; i++) >>>>> { >>>>> workder_t worker; >>>>> >>>>> worker.base = event_base_new(); >>>>> if (worker.base == NULL) >>>>> return -1; >>>>> >>>>> worker.httpd = evhttp_new(worker.base); >>>>> if (worker.httpd == NULL) >>>>> return -1; >>>>> >>>>> worker.soc = evhttp_accept_socket_with_handle(worker.httpd, nfd); >>>>> >>>>> evhttp_set_gencb(worker.httpd, httpserver_GenericHandler, NULL); >>>> >>>> No such function -- httpserver_GenericHandler. >>>> >>>>> >>>>> r = pthread_create(&worker.thr, NULL, httpserver_Dispatch, >>>>> worker.base); >>>>> >>>>> if (r != 0) >>>>> return -1; >>>>> >>>>> _workers.push_back(worker); >>>>> } >>>>> >>>>> std::cout << "Running - press return to stop..." << std::endl; >>>>> getchar(); >>>>> std::cout << "Stopping..." << std::endl; >>>>> >>>>> close(nfd); >>>>> std::cout << "Closed listening socket..." << std::endl; >>>>> >>>>> for(auto worker = _workers.begin(); worker != _workers.end(); ++worker) >>>>> { >>>>> // I would expect that this would cause the thread blocked in >>>>> event_base_dispatch() function called in httpserver_Dispatch function to >>>>> end >>>>> event_base_loopbreak(worker->base); >>>>> >>>>> // seems like I should do this, but not sure…works either way >>>>> evhttp_free(worker->httpd); >>>> >>>> This will close accept socket, because of LEV_OPT_CLOSE_ON_FREE in >>>> evhttp_accept_socket_with_handle(), you must use evhttp_bind_listener() >>>> here to avoid this. >>>> >>>> You could look into libevhtp, it has thread support out-of-the-box, and >>>> it is simpler since it is library that just implements http server. >>>> >>>>> >>>>> // If I do this the the event_base_dispatch() does return so I get the >>>>> expected result, but this feels wrong to >>>>> // destroy the base before the thread ends >>>>> // comment the next line to see the problem >>>>> event_base_free(worker->base); >>>> >>>> Here you first freeing base, and then in separate thread >>>> event_base_dispatch() will use some internal fields in it after free -- >>>> IOW you must free all resources after pthread_join(). >>>> >>>>> >>>>> // We are in the main thread so we join with each worker thread here >>>>> pthread_join(worker->thr, NULL); >>>>> } >>>>> std::cout << “Closed..." << std::endl; >>>> >>>> Bad quotes. >>>> >>>>> >>>>> return 0; >>>>> } >>>>> >>>>> int main(int argc, char* argv[]) >>>>> { >>>> >>>> In short all you need is add the next line here: >>>> evthread_use_pthreads(); >>>> >>>> This will enable inter-thread notification for loop break. >>>> >>>> [ I do have some other changes to you program, so if just adding >>>> evthread_use_pthreads() didn't work for you -- let me know, I will send >>>> you patch with my local changes ]. >>>> >>>>> httpserver_start(8080, NUM_THREADS, 100); >>>> >>>> NUM_THREADS undefined. >>>> *********************************************************************** >>>> To unsubscribe, send an e-mail to [email protected] with >>>> unsubscribe libevent-users in the body. >>> >>> *********************************************************************** >>> To unsubscribe, send an e-mail to [email protected] with >>> unsubscribe libevent-users in the body. >> >> -- >> Respectfully >> Azat Khuzhin > > -- > Respectfully > Azat Khuzhin > *********************************************************************** > To unsubscribe, send an e-mail to [email protected] with > unsubscribe libevent-users in the body. *********************************************************************** To unsubscribe, send an e-mail to [email protected] with unsubscribe libevent-users in the body.
