On Fri, Nov 21, 2014 at 4:48 PM, Justin Lloyd <[email protected]> wrote:
> Yes, if a relay becomes unavailable, metrics from certain hosts would get > lost until the relay was restored. It's not ideal but it's an improvement > over our current architecture. This is in AWS and I'm using Salt for > configuration management, so ideally I'd be able to spin up a replacement > relay pretty quickly in the event of a complete relay failure. > > My goal is to lose as few metrics as possible (a single poll-cycle or two) during an infrastructure blip, so the receiving layer must be redundant/resilient. > I am still considering a possible way to use a load balancer (ELB since > this is in AWS) but, as I mentioned, I have other considerations that may > limit my ability to do that. > FWIW, I'm using ELB to front pairs of relays, and have an issue I haven't yet been able to solve where restarting/stopping one of the relays does not lead to a graceful reconnection of clients; they remain stuck in CLOSE_WAIT for a while, until restarted. There is apparently a brief history of other services fronted with ELBs with clients ending up in this state, so I'm still looking for a fix and/or alternate configurations. -tt > > > On Fri, Nov 21, 2014 at 1:11 PM, Tom Throckmorton <[email protected]> > wrote: > >> On Fri, Nov 21, 2014 at 3:36 PM, Justin Lloyd <[email protected]> wrote: >> >>> Load balancing and/or round-robin DNS were not appropriate for our >>> situation given other architectural and application considerations in our >>> environment. >>> >>> My solution turned out to be simpler than I expected once I put two and >>> two together with the muti-node capability of the write_graphite plugin >>> combined with a PostCacheChain, like >>> https://gist.github.com/anonymous/b3c1cd692632216c4c35. >>> >> >> >> Ah, cool - like this -> >> https://collectd.org/wiki/index.php/Match:Hashed/Config So you're >> making collectd send some metrics to one relay, some to another etc. What >> happens when one of the relays goes unavailable? Not trying to poke holes, >> but to understand your viewpoint - couple of ways to achieve >> distribution/redundancy here, and always good to hear other ideas... >> >> -tt >> >> >> >> >>> >>> >>> >>> >>> >>> On Thu, Nov 20, 2014 at 6:26 PM, Tom Throckmorton <[email protected]> >>> wrote: >>> >>>> On Thu, Nov 20, 2014 at 2:33 PM, Justin Lloyd <[email protected]> wrote: >>>> >>>>> Hey all, >>>>> >>>>> I'm looking for a way to have Collectd be able to send to a set of >>>>> Graphite relays. Basically right now I have a three-layered Graphite >>>>> setup, >>>>> with a layer of four relays (for load distribution and server redundancy) >>>>> sending to a layer of four aggregators that send to a set of carbon-cache >>>>> processes on the cache server. I need my Collectd agents to be able to >>>>> send >>>>> to any of the four relays but I can't figure out how to accomplish this. >>>>> >>>> >>>> Hi Justin; >>>> >>>> If you want clients to truly send to *any* of the relay nodes, then >>>> maybe you want to look at fronting your relays with load-balancing, or >>>> maybe even round-robin DNS? Each have their benefits and tradeoffs, of >>>> course, but in either case the collectd config is as simple as having a >>>> single relay destination. >>>> >>>> >>>>> I was hoping the filtering >>>>> <http://collectd.org/documentation/manpages/collectd.conf.5.shtml#filter_configuration> >>>>> capability >>>>> would help but I don't see a way to specify a relay-specific >>>>> write_graphite >>>>> plugin per rule. >>>>> >>>> >>>> In that case, would you be sending to *all* of the relays? (vs. any) >>>> >>>> -tt >>>> >>>> >>>>> >>>>> Any suggestions? I did come across backstop >>>>> <https://github.com/obfuscurity/backstop> but I didn't want to >>>>> introduce a Rack app into the mix. >>>>> >>>>> Thanks, >>>>> Justin >>>>> >>>>> >>>>> _______________________________________________ >>>>> collectd mailing list >>>>> [email protected] >>>>> http://mailman.verplant.org/listinfo/collectd >>>>> >>>>> >>>> >>> >> >
_______________________________________________ collectd mailing list [email protected] http://mailman.verplant.org/listinfo/collectd
