JVM based there are lots of examples online, but something like this: https://www.baeldung.com/httpclient-connection-management Where you set the maximum number of outbound connections to some limited number. If you have more bolts trying to execute requests against a pool configured with fewer connections, your bolts would block until a connection became available, essentially throttling? Note: This breaks down once you have bolts running on multiple JVMs, as each JVM would have its own connection pool.
For external proxy service, which would work better when you have multiple JVMs involved, you could use something like the apache proxy module <https://stackoverflow.com/questions/14493574/configure-apache-web-server-to-use-a-proxy-server>. I want to say it's possible to configure apache to restrict the number of outbound connections to a specific set of hosts. There may be better software to solve this, it's just what jumps to mind for me. On Mon, Nov 25, 2019 at 4:41 PM Gowtham S <[email protected]> wrote: > *Could you make use of an outbound HTTP connection pool? (FYI: Our bolts > are span in multiple JVMs).* > > Is there any tool available for this? Can you please suggest if any > > With regards, > Gowtham S > > > On Mon, 25 Nov 2019 at 12:47, Stephen Powis <[email protected]> wrote: > >> Could you make use of an outbound http connection pool? Either via an >> external proxy service your bolts talk through.... or some JVM based http >> connection pool (which might get tricky if your bolts span multiple JVMs). >> >> On Mon, Nov 25, 2019 at 3:48 PM Gowtham S <[email protected]> >> wrote: >> >>> Hi, >>> >>> >>> We have certain bolts that invokes a few web services. However, those >>> endpoints have limited throughput. So we wanted to find out if there any >>> recommendations on how to throttle the calls so that they don't overload >>> the downstream services. >>> >>> Please let me know if there any hooks available in Storm, what are the >>> patterns that can be used and what are the best practices/pitfalls for >>> using them. >>> >>> Thanks and regards, >>> Gowtham S, MCA >>> PH: 9597000673 >>> >>
