On Sat, Jan 03, 2015 at 08:41:31PM +0300, Azat Khuzhin wrote: > On Sat, Jan 03, 2015 at 05:02:36PM +0000, Gerry Sweeney wrote: > > Hi Azat, > > > > Thank you for the detailed response and sorry for the late reply, I do > > really appreciate your time to respond - thank you. May I also appologize > > for the incompleteness of the example, I lifted the code out of a bigger > > project and failed to compile it stand-alone before I sent it - lesson > > learned :) > > > > Yes that fixed the problem, it also highlighted another problem, I was > > linking against an earlier version of the lib event library so the > > evthread_use_pthreads() could not be resolved at link time, I have also > > corrected that problem and also used “event2/event.h” instead of “event.h”. > > It is not that obvious from the various examples that there are different > > headers and library versions to contend with (thats my excuse anyway :) > > event.h is just a deprecated wrapper for event2/*, so it must works with > event.h too (But *yes, it is deprecated, so please don't use it*). > > > > > It now works as expected so thank you for your help. I did try to use > > libevhtp but I could not get it to compile, I think this was most likely > > down to the fact I was including version 1x of the lib event, I will try it > > again. > > AFAIR libevhtp needs libevent 2. > > > > > I am trying to create two different HTTP server patterns: - > > > > First one is a simple multi-threaded web server, where I receive requests > > asynchronously complete HTTP requests, event driven into a buffer (so > > presumably a buffered event) - this is the async part. Once I receive a > > valid HTTP request from the client, I put the request (and its connection) > > into a queue to be serviced synchronously by a thread from a pool of worker > > threads, ones the request has been served, the connection is once again put > > into the async receive queue ready for the next request. > > > > The second pattern, I want to create a server to be able to accept a large > > number of long-term and generally idle connections. Like the first example, > > the event driven receive will handle the request part, and once a valid > > request has been received the connection will be added to a connection > > list. At any point, other threads in the server will be able to send data > > to any clients open connection by looking up the open connection in the > > connection list. To visualise in HTTP terms the response data would most > > likely look like a never-ending chunked response, each chunk would be sent > > as times dictated by the server system. > > I don't see *big* difference between this two patterns (but maybe I > missed something?): > 1) accept() -> do job in a thread -> wait for other requests from client > 2) accept() -> do job in a thread (send chunked responses forever) > > And libevhtp works just like this, it accept() in the main thread, and > after this schedule callbacks into separate thread pool, by using pipes > (pipe(2)).
Sorry, not pipes, but sockets (i.e. socketpair(2)) > > Here is brief explanation how this part works in libevhtp: > - evhtp.c: > _evhtp_accept_cb() -> evthr_pool_defer() > - evthr.c: > evthr_pool_defer() -> evthr_defer() -> send() -> ... -> _evthr_read_cmd() > > And _evthr_read_cmd() works in a separate thread, while send() in main. > > And please could you avoid top-posting? > > Cheers, > Azat. > > > > > I am sure there are more formal names/descriptions for these patterns but > > thats as I understand them. These patterns are both terribly important for > > web-scale application design so they are worthy of some good documentation > > I think. > > > > For me personally, I thought it would be good to learn how to do this using > > libevent just as an exercise to learn how the library and event model > > implemented by libevent works, but so far its not been terribly obvious how > > I should approach either of these. I tried the first pattern but failed to > > make it work, evthread_use_pthreads() might solve this now so I will need > > to test. > > > > It would be terribly helpful if someone could point me in the right general > > direction with some pseudo code, which functions I would call to achieve > > those patterns, I know that may be asking a bit much but I thought I would > > ask anyway. > > > > For what its worth, I will make an effort do document my efforts on these > > two patterns on my blog http://gerrysweeney.com/ in order to hopefully > > feedback to the community. > > > > Gerry > > > > > > > > > > > Gerry Sweeney > > http://gerrysweeney.com/ > > > > > > > On 30 Dec 2014, at 16:00, Azat Khuzhin <[email protected]> wrote: > > > > > > On Sun, Dec 28, 2014 at 06:49:28PM +0000, Gerry Sweeney wrote: > > >> Hi Azat, > > >> > > >> See below, this is the program, see the comments in main() for an > > >> explanation of the problem. This needs to be a c++ compile as I am using > > >> an STL <vector>. I have made it work by freeing the event_base object > > >> by calling free_event_base() but that feels like the wrong thing to do. > > >> > > >> Thanks for the response, I appreciate it…any help would be much > > >> appreciated > > > > > > Please see comments for code, in short all you need in > > > evthread_use_pthreads(), but you code have some design issues and also > > > errors that don't allow me to compile this simple program. > > > > > >> > > >> Gerry > > >> > > >> ---------------------------------- > > >> #include <sys/types.h> > > >> #include <sys/time.h> > > >> #include <sys/queue.h> > > >> #include <stdlib.h> > > >> #include <unistd.h> > > >> #include <err.h> > > >> #include <event.h> > > >> #include <evhttp.h> > > >> #include <fcntl.h> > > >> #include <sys/socket.h> > > >> #include <netinet/in.h> > > >> #include <iostream> > > >> #include <vector> > > >> > > >> void httpserver_ProcessRequest(struct evhttp_request *req) { > > >> struct evbuffer *buf = evbuffer_new(); > > >> if (buf == NULL) > > >> return; > > >> > > >> evbuffer_add_printf(buf, "Requested: %s", evhttp_request_uri(req)); > > >> > > >> evhttp_send_reply(req, HTTP_OK, "OK", buf); > > >> > > >> evbuffer_free(buf); > > >> } > > >> > > >> void* httpserver_Dispatch(void *arg) { > > >> > > >> event_base_dispatch((struct event_base*)arg); > > >> > > >> std::cout << "Event dispatch thread ended..." << std::endl; > > >> > > >> return 0; > > >> } > > >> > > >> int httpserver_bindsocket(int port, int backlog) > > >> { > > >> int r; > > >> int nfd; > > >> nfd = socket(AF_INET, SOCK_STREAM, 0); > > >> if (nfd < 0) return -1; > > >> > > >> int one = 1; > > >> r = setsockopt(nfd, SOL_SOCKET, SO_REUSEADDR, (char *)&one, > > >> sizeof(int)); > > >> > > >> struct sockaddr_in addr; > > >> memset(&addr, 0, sizeof(addr)); > > >> addr.sin_family = AF_INET; > > >> addr.sin_addr.s_addr = INADDR_ANY; > > >> addr.sin_port = htons(port); > > >> > > >> r = bind(nfd, (struct sockaddr*)&addr, sizeof(addr)); > > >> if (r < 0) return -1; > > >> r = listen(nfd, backlog); > > >> if (r < 0) return -1; > > >> > > >> int flags; > > >> if ((flags = fcntl(nfd, F_GETFL, 0)) < 0 > > >> || fcntl(nfd, F_SETFL, flags | O_NONBLOCK) < 0) > > >> return -1; > > >> > > >> return nfd; > > >> } > > >> > > >> int httpserver_start(int port, int nthreads, int backlog) > > >> { > > >> int r, i; > > >> int nfd = httpserver_bindsocket(port, backlog); > > >> if (nfd < 0) > > >> return -1; > > >> > > >> struct workder_t > > >> { > > >> struct event_base* base; > > >> struct evhttp* httpd; > > >> struct evhttp_bound_socket* soc; > > >> pthread_t thr; > > >> }; > > >> > > >> std::vector<workder_t> _workers; > > >> > > >> for (i = 0; i < nthreads; i++) > > >> { > > >> workder_t worker; > > >> > > >> worker.base = event_base_new(); > > >> if (worker.base == NULL) > > >> return -1; > > >> > > >> worker.httpd = evhttp_new(worker.base); > > >> if (worker.httpd == NULL) > > >> return -1; > > >> > > >> worker.soc = evhttp_accept_socket_with_handle(worker.httpd, nfd); > > >> > > >> evhttp_set_gencb(worker.httpd, httpserver_GenericHandler, NULL); > > > > > > No such function -- httpserver_GenericHandler. > > > > > >> > > >> r = pthread_create(&worker.thr, NULL, httpserver_Dispatch, > > >> worker.base); > > >> > > >> if (r != 0) > > >> return -1; > > >> > > >> _workers.push_back(worker); > > >> } > > >> > > >> std::cout << "Running - press return to stop..." << std::endl; > > >> getchar(); > > >> std::cout << "Stopping..." << std::endl; > > >> > > >> close(nfd); > > >> std::cout << "Closed listening socket..." << std::endl; > > >> > > >> for(auto worker = _workers.begin(); worker != _workers.end(); > > >> ++worker) > > >> { > > >> // I would expect that this would cause the thread blocked in > > >> event_base_dispatch() function called in httpserver_Dispatch function > > >> to end > > >> event_base_loopbreak(worker->base); > > >> > > >> // seems like I should do this, but not sure…works either way > > >> evhttp_free(worker->httpd); > > > > > > This will close accept socket, because of LEV_OPT_CLOSE_ON_FREE in > > > evhttp_accept_socket_with_handle(), you must use evhttp_bind_listener() > > > here to avoid this. > > > > > > You could look into libevhtp, it has thread support out-of-the-box, and > > > it is simpler since it is library that just implements http server. > > > > > >> > > >> // If I do this the the event_base_dispatch() does return so I get the > > >> expected result, but this feels wrong to > > >> // destroy the base before the thread ends > > >> // comment the next line to see the problem > > >> event_base_free(worker->base); > > > > > > Here you first freeing base, and then in separate thread > > > event_base_dispatch() will use some internal fields in it after free -- > > > IOW you must free all resources after pthread_join(). > > > > > >> > > >> // We are in the main thread so we join with each worker thread here > > >> pthread_join(worker->thr, NULL); > > >> } > > >> std::cout << “Closed..." << std::endl; > > > > > > Bad quotes. > > > > > >> > > >> return 0; > > >> } > > >> > > >> int main(int argc, char* argv[]) > > >> { > > > > > > In short all you need is add the next line here: > > > evthread_use_pthreads(); > > > > > > This will enable inter-thread notification for loop break. > > > > > > [ I do have some other changes to you program, so if just adding > > > evthread_use_pthreads() didn't work for you -- let me know, I will send > > > you patch with my local changes ]. > > > > > >> httpserver_start(8080, NUM_THREADS, 100); > > > > > > NUM_THREADS undefined. > > > *********************************************************************** > > > To unsubscribe, send an e-mail to [email protected] with > > > unsubscribe libevent-users in the body. > > > > *********************************************************************** > > To unsubscribe, send an e-mail to [email protected] with > > unsubscribe libevent-users in the body. > > -- > Respectfully > Azat Khuzhin -- Respectfully Azat Khuzhin *********************************************************************** To unsubscribe, send an e-mail to [email protected] with unsubscribe libevent-users in the body.
