Matt, if it is a file server, then you should do it asynchronously.... totally asynchronously with async IO. That way threads are not a problem and the fair share algorithms already built into TCP/IP and the OS will do all the work for you.
So you startAsync, register a WriteListener and then Jetty will call you back with an onWritePossible call whenever a write can be done without blocking. You will then have to use async IO to actually read your content off disk and then write some. Look at our AsyncProxyServlet for an example... but replace to HttpClient with a File IO API. This can be a bit more complex code than you'd think.... but it is absolutely the right way to server files (or file like content). cheers On Mon, 1 Jun 2020 at 16:34, Matthew Boughen <[email protected]> wrote: > Hi Greg > > Thank you very much for your suggestions, they're really useful. > > Yes - I was thinking that we need to limit the total number of threads, as > each of them can use a lot of memory. I guess we could increase the thread > pool size to something large, and put a fair semaphore around the actual > work. After 50ms work is done, a startAsync followed by an immediate > redispatch would, I think, preserve the behaviour of ThreadLimitHandler - > particularly that requests from a single client remain interleaved, due to > the FIFO queue per Remote. (correct me if I'm wrong!) > > I should also have mentioned that writing the response (and reading from > disk) is much of the work of each 50ms chunk. I guess it's something like > a file server. So the clients (which are generally not browsers) aren't > waiting minutes for the HTTP headers, and we need to stream the response > (as they can be very large). > > With the concurrent queue idea, I guess after each chunk of work, you > could always add yourself to the back of the queue and dispatch a request > from the front (which could be yourself - in that case it doesn't matter > that you're hogging a thread). You'd also need to reserve one thread to > accept new requests, which are immediately put on the queue (so no blocking > work is done). (I think you'd also need a reserved thread in your > suggestion?) > > Agreed that (assuming you agree with it) the non-async approach is > probably better. (It's also clearly better than a separate thread pool!) > > Best > > Matt > > ------------------------------ > *From:* [email protected] <[email protected]> > on behalf of Greg Wilkins <[email protected]> > *Sent:* 01 June 2020 13:31 > *To:* JETTY user mailing list <[email protected]> > *Subject:* Re: [jetty-users] Interleaving requests for > fairness/starvation? > > Matt, > > Are you doing this to try to limit the total threads, or just to make sure > that some threads are not starving others? If you don't mind the total > number of threads, then you could simply put in a Thread.yield() call every > 50ms and not worry about async servlet handling. > > Note also, from a users perspective, having requests that can take many > minutes to respond is not great. If your clients are browsers, then it > could be better to separate your handling from the request/response thread > - ie start the handling and then immediately respond. Your clients could > then poll (or better yet long poll) to see progress and to get the final > state. > > I'm very cautious about any approach that changes the behaviour of the > threadpool, especially tryExecute. The behaviour there is all very finely > tuned to our scheduling strategies to avoid deadlocks and thread starvation. > > If you do want to use servlet async to do the interleaving, you need to > get other requests/threads to do the dispatching. EG every 50ms of > processing your app can look at a concurrent list to see if other processes > are waiting to be handled, if they are, then dispatch the first of them > then start async and add your AsyncContext to the end of the queue. The > problem with this approach is how does the first request ever get added to > the queue? Perhaps you need a limit on concurrent handling as well and if > you are above that, add yourself to the queue... if you finish handling and > reduce the handling count below the threshhold, then you can also wakeup > any waiting contexts on the queue. This is making Thread.yield feel > very simple. > > cheers > > > > > > On Mon, 1 Jun 2020 at 13:51, Matthew Boughen <[email protected]> wrote: > > Hi > > I was wondering if anyone could confirm whether what we've done is > sensible - it seems to work, but I'm not really confident and would like to > know how righteous it is! > > Our HTTP server processes requests that can take minutes at a time, the > handling for which can easily be broken down into small chunks. We would > like to interleave the handling of these requests so that none get starved > and time out (and to help with fairness). For each request, after 50ms of > handling, we asynchronously suspend the request and (hope to) redispatch it > to the queue behind newly arrived requests / other more-recently suspended > requests. > > With this approach, the issue seems to be in choosing where to call > asyncContext.dispatch: too close to the suspend call (startAsync) and Jetty > will not release the thread back to allow processing of other queued > requests (presumably because Jetty thinks it's more efficient to avoid the > context switch and so immediately continues processing the current > request). > > We've currently settled on overriding QueuedThreadPool.execute(Runnable) > and tryExecute(Runnable) to wrap the Runnable: any request that had > registered itself while the original (unwrapped) Runnable was called is now > redispatched. This seems to work in that: (a) only one request attempts to > register itself each time a Runnable is run, (b) the redispatch normally > seems to happen very quickly after the request has registered itself, and > (c) the request handling is indeed interleaved. > > Does that sound like the right thing to do here? I'm thinking it would be > as long as each request handle is guaranteed to be the tail call of a task > submitted to the thread pool (I don't know enough about the use of > ExecutionStrategy to know this). > > Any suggestions/comments are highly appreciated. > > Matt > > > P.S. I'm anticipating that the right thing to do might be to use a > separate thread pool and manage the interleaving there? This of course would > incur another thread context switch / managing another pool. We're also > using ThreadLimitHandler to avoid single clients taking out the whole > server, so would need to duplicate that logic. > > _______________________________________________ > jetty-users mailing list > [email protected] > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > > > > -- > Greg Wilkins <[email protected]> CTO http://webtide.com > _______________________________________________ > jetty-users mailing list > [email protected] > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > -- Greg Wilkins <[email protected]> CTO http://webtide.com
_______________________________________________ jetty-users mailing list [email protected] To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
