On 03/11/2015 12:58 PM, [email protected] wrote: > Hi Christian, > > I wanted to use external threads because my system has the following > constraints: > 1) One connection per thread: the response to the client is time critical and > I cannot afford that a connection delays another one.
Then you should use the one-thread-per-connection mode, that is exactly what you want. MHD will use a thread for each connection, which matches your requirements. This method also has no limits on open FDs as MHD doesn't use select() but blocking read/write operations instead. > 2) Many listening sockets are opened, hence many MHD daemon: some for HTTP > and some for HTTPS. > 3) Up to 2000 concurrent connections (shared between HTTP and HTTPS). Well, 2000 isn't much, so one-thread-per-connection should be fine. > As I don't know how the traffic will be shared amongst the daemons, I would > need to configured each MHD daemon with epoll and 2000 threads. > This solution does not scale. Just imagine the number of threads created at > start up or debugging a single HTTP request with gdb. 2000 threads is fine on modern systems, and if you really run into problems, you can easily switch to thread-pool-with-epoll. As for debugging, well, that's always an issue if you have 2000 concurrent actions ;-). > The best solution would be to share the worker threads pool between the > daemons but if I understand correctly the current implementation does not > allow it. Well, if you use on thread per connection, there is no sharing anyway. If you have HTTP and HTTPS, you are right right now you'd have two daemons and no joint pool. Two solutions: you can setup a reverse proxy using Apache or nginx to forward HTTPS to HTTP, or just live with a slight imbalance. I suspect the reverse proxy is the best option, as then you can even move that to another physical machine easily, and you're off-loading HTTPS. > This why I fall back on external thread: I create 2000 daemons for each > listening socket and dispatch the HTTP/HTTPS request to a pool of 2000 > threads, which reuse the daemons. > > Is there a plan to implement worker threads sharing between daemons? If not, > are there other MHD solutions that I miss? I don't have a plan to do that, seems a bit overengineered. For this stuff, I'd rather keep it simple unless given hard proof that this is a major issue. I don't think without having performance measurements one should even worry about this. Happy hacking! Christian > Thanks, > > Louis > > Le 05/03/15, Christian Grothoff <[email protected]> a écrit : >> Hi! >> >> I had no plans for an 'external poll' API, as we have epoll() which is >> better for very large FDsets (as long as you're on Linux). Also, >> redefining FD_SETSIZE is also IMO a very good possibility. >> >> Regardless, select/poll with > 1024 FDs and external mode is likely not >> the greatest idea from a scalability/performance perspective: if you >> have that many concurrent / open FDs, you really want epoll() and >> possibly think about using threads / thread pool for processing. >> >> Happy hacking! >> >> Christian >> >> On 03/04/2015 10:22 PM, [email protected] wrote: >>> Hi, >>> >>> I would like to use the external select mode, as I need to manage the >>> connection threads externally. >>> Because there could be much more than 1024 FDs used at run time, I cannot >>> use MHD_get_fdset() and select (). >>> Apart from redefining FD_SETSIZE, are there other ways to use external >>> threads when FD > 1024? >>> If not, is there a plan to add an MHD_get_poll_event API, the poll version >>> of MHD_get_fdset()? >>> >>> Thanks, >>> >>> Louis Benoit >>> >> >> >> >
