Yeah that's one area Java has always been a little weak with - I know of none Java systems where you can monitor requests and if one is taking much too long or two many resources, you can just hard kill it. I'm not sure there is much you can do java that's less ugly or more effective than just haveing spots along the request path where the thread checks in itself whether it should surrender.
I suppose having various stages in the request where you check in is not the worst strategy though - it could even work like the rate limiting should when requests first come in, and instead of failing immediately, use the async webapp api to suspend the request, give up the thread and save the context. Then if room opens up fast enough, the request is resumed and if not, it's rejected. On Sun, Jul 23, 2023 at 4:39 AM Ilan Ginzburg <ilans...@gmail.com> wrote: > I also think we should gate at the entry point (decide to process a > request, delay its processing or reject it) then do it as fast as possible > (and potentially decide to reject and abandon during processing if the > realization that it should have been rejected happens too late). > Slowing down processing under high load is going to increase the load (more > in flight stuff) plus the usual priority inversion issue (not starvation) > given we're reasonably not going to preempt. > > Ilan > > > On Sun, Jul 23, 2023, 5:35 AM Mark Miller <markrmil...@gmail.com> wrote: > > > I think the problem is that it’s just extremely difficult to reason > about. > > I think that’s really the root of the concern, and he’s generally working > > with code that’s likely close to hardware ideal - a thread count not much > > more than the number of cores at the high end. Here you have to reason > > about > > hundreds to thousands of threads with a very large number of sources. > > > > And then rate limiting is tough to stack on to that reasoning. It’s not > > going to be nicely tied to the hardware capabilities and capacity - it’s > a > > static throttle on a very dynamic environment. Netflix actually has a > very > > interesting open source project on GitHub that tries to solve that > problem > > with some clever math and dynamic limits. Even that is a bandaid on a > very > > hard problem though, and it’s all targeted at requests, and there is > plenty > > of room for threads having liveness, deadlock, and starvation issues that > > are not so neatly tied to requests. > > > > And really, IMO if you solve good throttling at the point of incoming > > requests > > (dynamic limits reflective of current resource usage, capacity, traffic, > > minimal client retry kick backs that occur when truly over load and not > > around optimal max throughout, with wait queues that don’t sit on threads > > or more than nominal resources) > > prioritizing or holding back threads handling in progress threads becomes > > not very appealing and much more likely to hamper peak/good throughout > due > > to > > the locky nature of Java and the way Solr requests can depend on each > > other. I think if you are in that situation, there’s such a risk of > making > > things worse, that it’s best to let every in flight request race it’s way > > through as fast as it can while you just efficiently gate new request > from > > starting. > > > > The whole situation is really one of the largest downsides to Java. Java > > puts > > you on a downhill road to a system that abhors approaching, say nothing > > about maintaining, optimal max throughput. And so you generally have > death > > spiral limits or artificially low static limits with poor overload > behavior > > and human operator time reaction. If there even is a win in that world > > around thread priorities, the risk reward calculation is just that good. > > > > The only real good thing I’ve ever seen, other than a hand wavy mythical > > promise of the always coming green threads, is a fully async system with > > effectively built in back pressure. > > > > I’m just tossing out prattle though, not arguing against thread > priorities. > > Some projects have tried them out here or there. I think they commonly > end > > up removed and commonly don’t get credited with a noticiable impact, but > > implementation, testing, and results always beats advice, best practices > > and warnings. > > > > On Sat, Jul 22, 2023 at 5:39 PM David Smiley <david.w.smi...@gmail.com> > > wrote: > > > > Thanks. I could see thread priority customization being used well in > > combination with rate limiting so as to mitigate a starvation risk. > > > > > > > > > Yeah, I met Brian Goetz and have his excellent book. > > > > > > > > > ~ David > > > > > > > > > > > > On Sat, Jul 22, 2023 at 3:20 AM Mark Miller <markrmil...@gmail.com> > wrote: > > > > > > > > > > It’s a hint for the OS, so results can vary by platform. Not the end of > > the > > > world but not ideal. > > > > > > A scarier fact is that Brian Goetz, pretty big name in Java > concurrency, > > > recommends against in general, noting that it can lead to liveness / > > > starvation issues. > > > > > > > > > > > -- - MRM
