Same here, running 3.6.5 for (some) of the rabbit clusters. It's been stable over the last month (fingers crossed!), though: * gave up on stats collection (set to 60000 which makes it not so useful) * can still make it very sick with a couple of misconfigured clients (rabbit_retry_interval=1 and rabbit_retry_backoff=60 currently everywhere).
Some data from the neutron rabbit cluster (3 vm nodes, not all infra currently talks to neutron): * connections: ~8k * memory used per node: 2.5GB, 1.7GB, 0.1GB (the last one is less used due to a previous net partition i believe) * rabbit hiera configuration rabbitmq::cluster_partition_handling: 'autoheal' rabbitmq::config_kernel_variables: inet_dist_listen_min: 41055 inet_dist_listen_max: 41055 rabbitmq::config_variables: collect_statistics_interval: 60000 reverse_dns_lookups: true vm_memory_high_watermark: 0.8 rabbitmq::environment_variables: SERVER_ERL_ARGS: "'+K true +A 128 +P 1048576'" rabbitmq::tcp_keepalive: true rabbitmq::tcp_backlog: 4096 * package versions erlang-kernel-18.3.4.4-1 rabbitmq-server-3.6.5-1 It's stable enough to keep scaling it up in the next couple months and see how it goes. Cheers, Ricardo On Mon, Jan 9, 2017 at 3:54 AM, Sam Morrison <sorri...@gmail.com> wrote: > We’ve been running 3.6.5 for sometime now and it’s working well. > > 3.6.1 - 3.6.3 are unusable, we had lots of issues with stats DB and other > weirdness. > > Our setup is a 3 physical node cluster with around 9k connections, average > around the 300 messages/sec delivery. We have the stats sample rate set to > default and it is working fine. > > Yes we did have to restart the cluster to upgrade. > > Cheers, > Sam > > > > On 6 Jan 2017, at 5:26 am, Matt Fischer <m...@mattfischer.com> wrote: > > MIke, > > I did a bunch of research and experiments on this last fall. We are running > Rabbit 3.5.6 on our main cluster and 3.6.5 on our Trove cluster which has > significantly less load (and criticality). We were going to upgrade to 3.6.5 > everywhere but in the end decided not to, mainly because there was little > perceived benefit at the time. Our main issue is unchecked memory growth at > random times. I ended up making several config changes to the stats > collector and then we also restart it after every deploy and that solved it > (so far). > > I'd say these were my main reasons for not going to 3.6 for our control > nodes: > > In 3.6.x they re-wrote the stats processor to make it parallel. In every 3.6 > release since then, Pivotal has fixed bugs in this code. Then finally they > threw up their hands and said "we're going to make a complete rewrite in > 3.7/4.x" (you need to look through issues on Github to find this discussion) > Out of the box with the same configs 3.6.5 used more memory than 3.5.6, > since this was our main issue, I consider this a negative. > Another issue is the ancient version of erlang we have with Ubuntu Trusty > (which we are working on) which made upgrades more complex/impossible > depending on the version. > > Given those negatives, the main one being that I didn't think there would be > too many more fixes to the parallel statsdb collector in 3.6, we decided to > stick with 3.5.6. In the end the devil we know is better than the devil we > don't and I had no evidence that 3.6.5 would be an improvement. > > I did decide to leave Trove on 3.6.5 because this would give us some bake-in > time if 3.5.x became untenable we'd at least have had it up and running in > production and some data on it. > > If statsdb is not a concern for you, I think this changes the math and maybe > you should use 3.6.x. I would however recommend at least going to 3.5.6, > it's been better than 3.3/3.4 was. > > No matter what you do definitely read all the release notes. There are some > upgrades which require an entire cluster shutdown. The upgrade to 3.5.6 did > not require this IIRC. > > Here's the hiera for our rabbit settings which I assume you can translate: > > rabbitmq::cluster_partition_handling: 'autoheal' > rabbitmq::config_variables: > 'vm_memory_high_watermark': '0.6' > 'collect_statistics_interval': 30000 > rabbitmq::config_management_variables: > 'rates_mode': 'none' > rabbitmq::file_limit: '65535' > > Finally, if you do upgrade to 3.6.x please report back here with your > results at scale! > > > On Thu, Jan 5, 2017 at 8:49 AM, Mike Dorman <mdor...@godaddy.com> wrote: >> >> We are looking at upgrading to the latest RabbitMQ in an effort to ease >> some cluster failover issues we’ve been seeing. (Currently on 3.4.0) >> >> >> >> Anyone been running 3.6.x? And what has been your experience? Any >> gottchas to watch out for? >> >> >> >> Thanks, >> >> Mike >> >> >> >> >> _______________________________________________ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators