Hi Jesse, I'm looking at BasicHttpFetcher and it seems like someone had this idea (or something similar) in mind when implementing it. There's logic in the fetch method to check for slow responses and call a protected slowResponseWarning method that currently only logs a warning if a response took too long. The comments on slowResponseWarning seem to suggest overriding it "if you want to do something other than logging a warning". From an implementation standpoint, you could keep track of slow responses via this method and then override the fetch method to check if the incoming request has been flagged as slow before calling super.fetch.
This still doesn't answer your question about how to best figure out what apps are slow when using a reverse proxy, but I think that's an issue that doesn't need to be solved in Shindig. It's pretty much implementation specific at that point. I don't think I really answered your question, but figured I would throw in my 2 cents anyways. :) Thanks, -Stanton From: "Ciancetta, Jesse E." <[email protected]> To: "[email protected]" <[email protected]>, Date: 04/25/2012 07:44 Subject: Shindig HttpFetcher Hi All, We're running into an issue with our HttpFetcher implementation where slow responses to outgoing HTTP requests are crashing our shindig server. Every once in a great while one of the services our gadgets call on for data will have some kind of issue and start returning really slowly which causes a backup of requests in the fetcher to the point that shindig starts to become unresponsive. So we're basically looking to implement some kind of dynamic throttling in our fetcher where we can detect that some service is having a problem and throttle down requests to that service until its response times return back to normal. However, one detail that makes our case a little trickier that normal (or maybe not depending on how common this practice is) is that we mask many different services behind a single host using a reverse proxy -- so all of these URL's on the same host may actually be hitting different servers on the backend: host.example.com/some-app host.example.com/some-collection-of-similar-apps/app-1/api/foo host.example.com/some-collection-of-similar-apps/app-1/api/foo/bar host.example.com/some-collection-of-similar-apps/app-1/api/foo/... host.example.com/some-collection-of-similar-apps/app-2 host.example.com/some-collection-of-similar-apps/app-2/foo host.example.com/different-app ... That makes truly dynamic throttling kind of tricky since the paths can vary arbitrarily -- so for example if this app was the one that was slow: host.example.com/some-collection-of-similar-apps/app-2 we want to throttle down requests to just that one app but let the others continue through. One other detail -- some of the apps behind our reverse proxy are legacy apps which are obnoxiously slow so we have to set the timeout on our fetcher to an obnoxiously high value to accommodate those legacy services which makes this problem just that much worse... The only solution I've come up with so far is to define some sort of regular expressions or something that we can use to figure out what app a given URL belongs to and then using that we can see that app-2 is slow and start throttling requests to it -- but that's a very manual/specific solution to the problem and we were hoping to come up with something more generic that we could push back into shindig core. Has anyone else has faced similar issues and/or have any suggestions on how to go about handling this? Thanks! --Jesse
