Hi All, We're running into an issue with our HttpFetcher implementation where slow responses to outgoing HTTP requests are crashing our shindig server. Every once in a great while one of the services our gadgets call on for data will have some kind of issue and start returning really slowly which causes a backup of requests in the fetcher to the point that shindig starts to become unresponsive.
So we're basically looking to implement some kind of dynamic throttling in our fetcher where we can detect that some service is having a problem and throttle down requests to that service until its response times return back to normal. However, one detail that makes our case a little trickier that normal (or maybe not depending on how common this practice is) is that we mask many different services behind a single host using a reverse proxy -- so all of these URL's on the same host may actually be hitting different servers on the backend: host.example.com/some-app host.example.com/some-collection-of-similar-apps/app-1/api/foo host.example.com/some-collection-of-similar-apps/app-1/api/foo/bar host.example.com/some-collection-of-similar-apps/app-1/api/foo/... host.example.com/some-collection-of-similar-apps/app-2 host.example.com/some-collection-of-similar-apps/app-2/foo host.example.com/different-app ... That makes truly dynamic throttling kind of tricky since the paths can vary arbitrarily -- so for example if this app was the one that was slow: host.example.com/some-collection-of-similar-apps/app-2 we want to throttle down requests to just that one app but let the others continue through. One other detail -- some of the apps behind our reverse proxy are legacy apps which are obnoxiously slow so we have to set the timeout on our fetcher to an obnoxiously high value to accommodate those legacy services which makes this problem just that much worse... The only solution I've come up with so far is to define some sort of regular expressions or something that we can use to figure out what app a given URL belongs to and then using that we can see that app-2 is slow and start throttling requests to it -- but that's a very manual/specific solution to the problem and we were hoping to come up with something more generic that we could push back into shindig core. Has anyone else has faced similar issues and/or have any suggestions on how to go about handling this? Thanks! --Jesse
