Hi Christian,
Thanks for the quick reply.
And erk! Sorry about the OS thing. I'm on Linux. My development
environment is a stock install of Ubuntu 12.04 LTS VM on a Windows 7 PC.
(I doubt the VM thing matters at all, but figured I'd mention it just in
case.)
Here's my MHD_start_daemon code, nothing earth shattering ( I assume that's
where the per-connection timeout would be set):
-----------------------
myDaemon = MHD_start_daemon(MHD_USE_THREAD_PER_CONNECTION, // one thread
per connection
WEB_TEST_PORT, // port to
listen on
nullptr, // accept
policy callback
nullptr, // extra arg
to accept policy callback
&connectionCallbackC, // main
'connection' callback
this, // extra
argument to the 'connection' callback
MHD_OPTION_NOTIFY_COMPLETED, // specifies
that the next arg is the callback to call when the connection is done
&requestCompletedCallbackC, // the
callback to call when the connection is done
this, // extra arg
to the callback when the connection is done
MHD_OPTION_CONNECTION_LIMIT, // specifies
that the next arg is the max number of simultaneous connections
(unsigned int)20, // the
number of permitted simultaneous connections
MHD_OPTION_END); // no more
options in the arg list
----------------------
So - no timeout changes for MHD. I haven't changed the default timeout for
the kernel, so I'm guessing that 115 (awfully close to 120, i.e., 2
minutes) is something else? I agree completely that it's likely some kind
of timeout, though.
I don't change any thread timeouts at all (although I have some other
threads in my application that I set to real-time priority for some audio
processing, but I can't imagine how that would adversely affect MHD).
If I didn't mention it before, I did trace this down to the 'join' of the
thread in 'close_all_connections'. I assume 'MHD_join_thread_' is actually
pthread_join since I'm on Linux. My gut was telling me that somehow the
thread isn't being told to stop as part of the shutdown process and is
eventually just timing out somewhere inside MHD (or via TCP as you
suggested).
Before I noticed this problem, I was running with
'MHD_USE_SELECT_INTERNALLY' with a thread-pool, but you had suggested in a
previous post (a few weeks ago) that that won't work properly with
'comet-like' requests unless I do the suspend/resume functionality. That's
perfectly fine and good, but it seems worth mentioning that I didn't notice
this problem when using that thread mode. From looking at the
'close_all_connections' code in daemon.c, I can see why the behavior could
be different.
Anyway - I'll play around with the HAVE_LISTEN_SHUTDOWN option, see if that
makes any difference. Failing that, I'll see if I can create a test case.
That make take a couple days, though - lots of code to strip out to keep it
simple. What should I do once I get a concise test case ready to go? Send
the code to the mailing list?
Thanks again for a great library - it's perfect for what I'm doing and I've
really enjoyed using it.
Ken
On Fri, Jun 6, 2014 at 5:52 PM, Christian Grothoff <[email protected]>
wrote:
> Dear Kenneth,
>
> The first thing you should provide is the name of the operating system
> you are on (if non-Linux, that might by itself be a big hint).
>
> Regardless, given that MHD does terminate after 115s suggests that some
> timeout is involved. Did you, by chance, set the per-connection timeout
> to like 115-120 seconds? Or is that your kernel's TCP timeout? Again,
> that's just for diagnostics -- there clearly still is an issue as that
> timeout should not matter for shutdown.
>
>
> When MHD is stopped, it should terminate existing ongoing requests
> immediately (client gets a closed TCP socket); this is usually done
> by calling 'shutdown()' on the TCP socket if HAVE_LISTEN_SHUTDOWN
> (which should unblock the pthread), or by writing to a pipe. Again,
> what happens exactly depends a lot on your thread mode, MHD options
> and kernel.
>
> However, shutdown issues like this are usually easy to debug by
> attaching a debugger and seeing where that thread hangs around,
> so if you could provide a testcase I'm sure we can find a solution
> rather quickly.
>
>
> Happy hacking!
>
> Christian
>
> On 06/06/2014 08:50 PM, Kenneth Mastro wrote:
> > I'm seeing an odd behavior when I shut down the daemon (MHD_stop_daemon).
> > The shutdown hangs for about 115 seconds, then it finally terminates
> > cleanly.
> >
> > I'm using the 'thread per connection' mode, which may be relevant. After
> > digging into microhttpd's code, I found that it was hanging on a thread
> > join during 'close_all_connections'.
> >
> > I began doing some heavy debugging, and determined that I don't have any
> > lingering connections that I never finished with. I even went so far as
> to
> > print out the thread PIDs in both microhttpd and my code to make sure I
> saw
> > the requests that were being handled by the hanging thread. Nothing
> fancy
> > - just some css and/or image files being served up (about 25 requests or
> > so). Through this debugging, I'm nearly certain that I'm ending the
> > connection/request callback every time (I also did a global
> > increment/decrement of requests, and it gets back to 0 every time - Ergo,
> > I'm not holding onto the thread.).
> >
> > Some other useful info:
> > * When I set the maximum number of connections/threads to 1 (instead of
> 20)
> > when I start the daemon, it does not hang when I stop the daemon - it
> just
> > ends immediately, as expected.
> >
> > * After I start shutting down, if I make ANOTHER request from the browser
> > (e.g.., refresh the page), the daemon stops immediately and cleanly.
> >
> > * After I load the page the first time (and it makes a bunch of requests
> to
> > the server, causing 1 or more threads to be used), if I wait a while
> (e.g.,
> > a minute) and then stop the server, it shortens the time it takes the
> > thread to terminate. I.e., if I wait 65 seconds, it only takes 70
> seconds
> > for the server to stop instead of 115. If I wait longer than 115
> seconds,
> > it will end immediately, as expected.
> >
> > * If I make NO requests (i.e., I just start, then stop a few seconds
> > later), it terminates cleanly. The 115 second delay only occurs after
> I've
> > made a request to the server from the browser and then try to stop it.
> >
> > * I'm using a fairly recent version: 0.9.35
> >
> >
> > I came across some vaguely related issues regarding stopping the daemon:
> > -----------------
> > I saw some old archived messages about a problem that existed in
> > microhttpd, but that was several years ago, but it sounds like it was a
> > full-on deadlock:
> > https://lists.gnu.org/archive/html/libmicrohttpd/2009-10/msg00013.html
> >
> > Also saw this, which may be similar to what I'm dealing with (only it
> > sounds like this was solaris specific, and I'm on Linux):
> > http://lists.gnu.org/archive/html/libmicrohttpd/2011-09/msg00024.html
> >
> > Also, this issue feels similar as well:
> > https://gnunet.org/bugs/view.php?id=1901
> > ------------------
> >
> > It seems like there's been some trouble in this area in the past, so I
> > figured it was worth asking about.
> >
> > So - any idea if I'm doing something wrong? I'm at a bit of a loss since
> > my code doesn't have any control over the internal threads. I'm happy to
> > try anything in my application and/or modify microhttpd to test, but I'm
> > not sure where to focus my efforts.
> >
> >
> > Thanks much,
> > Ken
> >
>
>
