----- Original Message ----- > From: "Christopher Wood" <[email protected]> > To: "mcollective-users" <[email protected]> > Sent: Wednesday, 19 October, 2016 09:31:58 > Subject: Re: [mcollective-users] translate activemq subcollectives to nats.io
> On Tue, Oct 18, 2016 at 11:32:29PM +0100, R.I.Pienaar wrote: >> Oh this is amazing news really good to hear about your performance metrics, >> >> did you deploy this using just the settings my module installs? > > I used the choria plugin only and then just configured the NATS connector. We > use my own derivation of the upstream mcollective module so there's no > mcollective::module_plugin type available. I still spent some time this > morning > troubleshooting mcollective clients despite my explanatory confluence page, > rather validating my approach of not going all-choria-at-once in this > environment. > > For background, I took a metaphorical chainsaw to the mcollective module > sometime in early 2015 and this drastically reduced our catalog resource count > with commensurate catalog compilation reduction time. This helped take some > strain off the pre-puppetserver apache+passenger setup. > > (I should probably publish what we use but need to find time to remove all the > private bits first.) > >> Can you tell me a bit about your memory and CPU usage in this case? > > I haven't measured quantitatively. > > There's still the same cpu spike with nats when I do something like "mco > package > status glibc -t 10", but it's a flash in the pan. I need to turn top down to a > 0.2 s or 0.5 s refresh interval to notice it. ActiveMQ would nail a core to > 100% for a noticeable interval. > > Today nats is using about 300 MB RAM whereas the (nearly empty) activemq > broker > on the same host is using 230 MB RAM. Yesterday when activemq was serving > mcollective daemons this broker (one of several) was using approx. 1.2 GB. > > Usual caveats: These are counting-on-my-fingers calculations based on % of RAM > as listed in "ps aux" and my recollection from top output yesterday. Somebody > who is better at ActiveMQ may have a better experience with it than me. Well that does sound good anyway, if you can look out for nodes that do not reconnect after interruption and let me know that would be helpful, I had this once but could not reproduce after Re monitoring, NATS has a monitoring port / service, if you use collectd you can use this to gather those metrics <Plugin "curl_json"> <URL "http://localhost:8222/varz";> Instance "gnatsd" <Key "connections"> Type "gauge" </Key> <Key "in_bytes"> Type "bytes" </Key> <Key "in_msgs"> Type "counter" </Key> <Key "max_control_line"> Type "gauge" </Key> <Key "max_payload"> Type "bytes" </Key> <Key "max_pending_size"> Type "bytes" </Key> <Key "mem"> Type "bytes" </Key> <Key "out_bytes"> Type "bytes" </Key> <Key "out_msgs"> Type "counter" </Key> <Key "remotes"> Type "gauge" </Key> <Key "routes"> Type "gauge" </Key> <Key "slow_consumers"> Type "gauge" </Key> <Key "subscriptions"> Type "gauge" </Key> <Key "total_connections"> Type "counter" </Key> </URL> </Plugin> > >> ----- Original Message ----- >> > From: "Christopher Wood" <[email protected]> >> > To: "mcollective-users" <[email protected]> >> > Sent: Tuesday, 18 October, 2016 14:49:51 >> > Subject: Re: [mcollective-users] translate activemq subcollectives to >> > nats.io >> >> > Good news, today we migrated our mcollective installation from ActiveMQ to >> > NATS. >> > There are the usual handful of straggler hosts but various departments have >> > tickets. I will switch my client back to activemq and check on the >> > leftovers in >> > a few days (clustered activemq brokers still running precisely for this >> > reason). >> > >> > It turned out that the security department was quite accomodating about >> > adding >> > $HOME/.puppetlabs to the "keep" list. They send frequent auditing queries >> > around and are getting faster, more consistent replies so they're happier >> > now. >> > ("All good for me too, after switching to it, I see speed improvements and >> > better reliability.") I see the same thing, queries where I would have had >> > to >> > extend the timeout with ActiveMQ are handled within my client's configured >> > "--dt 5 -t 5" with NATS. >> > >> > Finally, I will be able to turn off a few hosts. We're now handling 1900 >> > mcollective servers on a single unclustered NATS daemon where before we >> > needed >> > four clustered ActiveMQ daemons (I agree that my own ActiveMQ skill was a >> > likely factor here). The NATS daemon is using less cpu and ram than any >> > single >> > ActiveMQ daemon was. >> > >> > Now let's see how NATS handles its first DDOS or inter-datacenter >> > connection >> > downtime! Will follow up if I notice anything bad. >> > >> > On Thu, Oct 06, 2016 at 08:02:37PM +0100, R.I.Pienaar wrote: >> >> >> >> >> >> ----- Original Message ----- >> >> > From: "Christopher Wood" <[email protected]> >> >> > To: "mcollective-users" <[email protected]> >> >> > Sent: Thursday, 6 October, 2016 20:29:16 >> >> > Subject: Re: [mcollective-users] translate activemq subcollectives to >> >> > nats.io >> >> >> >> > On Thu, Oct 06, 2016 at 08:43:54AM +0100, R.I.Pienaar wrote: >> >> >> >> >> >> >> >> >> ----- Original Message ----- >> >> >> > From: "Christopher Wood" <[email protected]> >> >> >> > To: "mcollective-users" <[email protected]> >> >> >> > Sent: Wednesday, 5 October, 2016 22:19:50 >> >> >> > Subject: Re: [mcollective-users] translate activemq subcollectives >> >> >> > to nats.io >> >> >> >> >> >> > Subcollectives indeed work fine. So far I have things running on a >> >> >> > single host >> >> >> > and the setup was much easier than doing something similar with >> >> >> > activemq >> >> >> > several years ago. Everything I could think to try works the same. >> >> >> > >> >> >> > Next to fully puppetize the setup and persuade the security >> >> >> > department to open >> >> >> > another port for nats (oh dear). >> >> >> >> >> >> Did you use the module the wiki recommend? It's simple but does what >> >> >> you want >> >> >> including >> >> >> clustering >> >> >> >> >> >> > Mild nitpick: there doesn't seem to be a client config file option >> >> >> > for cert and >> >> >> > key. I know there are environment variable options but enough people >> >> >> > will >> >> >> > forget to set those on every login (environment with no persistent >> >> >> > .bash_profile or similar, yes indeed). My workaround will be to >> >> >> > provide a short >> >> >> > puppet manifest to reshape their client config since any shuffling >> >> >> > errors will >> >> >> > be obvious when mailed back. It would definitely be easier to have >> >> >> > people copy >> >> >> > activemq cert params into choria cert params. >> >> >> >> >> >> why do you wish to change these paths? It defaults to the same ones >> >> >> puppet use >> >> >> for a reason - you should not need to make any client configuration at >> >> >> all, just >> >> >> put the certs in the puppet location - which 'mco choria request_cert' >> >> >> does for >> >> >> you >> >> > >> >> > If I were setting things up from scratch in regular home directories >> >> > the choria >> >> > technique would be suitable, but I'm not. >> >> > >> >> > I have the usual corporate things going on: >> >> > >> >> > 1) The new cert location is not in "$HOME/.mcollective.d". It took a >> >> > great deal >> >> > of protracted discussion with the security department to permit >> >> > .mcollective >> >> > and .mcollective.d (certs and things) to persist overnight in users' >> >> > home >> >> > directories. Most every other file under $HOME on the authorized mco >> >> > client >> >> > hosts get removed nightly. I will be re-opening that discussion. >> >> >> >> OK, well adding configurable overrides for these would not be hard, I >> >> dont mind >> >> adding more flags, feel free to open a issue in the choria github >> >> >> >> > 2) Everybody has their own (working) certs and configs from the current >> >> > production install. It was like pulling teeth the first time to get >> >> > people set >> >> > up despite the detailed instructions; what is straightforward and >> >> > obvious to me >> >> > is not so obvious to others. That includes running a cert management >> >> > command >> >> > based on my experience here. >> >> >> >> reusing those would be hard right? There were never a CA involved in >> >> making >> >> those >> >> the new stuff does CA validation >> >> >> >> > NB: This is of course a policy issue not an mcollective issue, I myself >> >> > wouldn't >> >> > go and add this functionality based on the description of the issues. >> >> > Policies >> >> > have results and working around them in a corporate environment rarely >> >> > ends >> >> > well. >> >> >> >> no its fair enough, this stuff matters and mcollective is one of few in >> >> this >> >> space >> >> with solid AAA story because I want it to appeal to exactly this kind of >> >> user, >> >> so >> >> if overrides will help I will add them >> >> >> >> -- >> >> >> >> --- >> >> You received this message because you are subscribed to the Google Groups >> >> "mcollective-users" group. >> >> To unsubscribe from this group and stop receiving emails from it, send an >> >> email >> >> to [email protected]. >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > >> > --- >> > You received this message because you are subscribed to the Google Groups >> > "mcollective-users" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> > email >> > to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "mcollective-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email >> to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "mcollective-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email > to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
