Thanks. Yes, we do have schedulePeriodForDestinationPurge set.
On Thursday, 18 May 2017 13:50:11 UTC+1, Charlie Sharpsteen wrote:
> In addition to the policyEntries, you'll also need to set
> schedulePeriodForDestinationPurge on the top-level <broker> element ---
> otherwise no GC sweeps will occur to clean the inactive queues out. For
> <broker brokerName="pe-201645-master.puppetdebug.vlan" xmlns="
> http://activemq.apache.org/schema/core" persistent="false"
> If it is working, you'll see messages similar to the following in the
> activemq logs:
> 2017-05-04 22:05:48,850 | INFO |
> mcollective.reply.pe-201645-master.puppetdebug.vlan_10076.1 Inactive for
> longer than 300000 ms - removing ... |
> org.apache.activemq.broker.region.RegionBroker | ActiveMQ
> Broker[pe-20164nightly-master.puppetdebug.vlan] Scheduler
> More info:
> On Wednesday, May 17, 2017 at 9:15:06 PM UTC-7, Richard Houck wrote:
>> Do you have Reply Queue Pruning enabled?
>> I had the same problem until I configured this. Hope this helps.
>> MCollective sends replies on uniquely-named, single-use queues with
>> names like mcollective.reply.<UNIQUE ID>. These have to be deleted after
>> about five minutes, lest they clog up ActiveMQ’s available memory. By
>> default, queues live forever, so you have to configure this.
>> Use a <policyEntry> element for *.reply.> queues, with
>> gcInactiveDestinations set to true and inactiveTimoutBeforeGC set to
>> 300000 ms (five minutes).
>> <policyEntry queue="*.reply.>" gcInactiveDestinations="true"
>> inactiveTimoutBeforeGC="300000" />
>> Try the folloiwng:
>> <policyEntry queue=">" producerFlowControl="true" memoryLimit="20mb"
>> gcInactiveDestinations="true" inactiveTimoutBeforeGC="10000">.
>> Desired settings:
>> <policyEntry queue="*.reply.>" producerFlowControl="true"
>> memoryLimit="20mb" gcInactiveDestinations="true"
>> On Wednesday, May 17, 2017 at 7:09:32 AM UTC-7, Steven Meunier wrote:
>>> Hello all
>>> We're using MCollective with ActiveMQ but are running into problems with
>>> stability of the ActiveMQ cluster. The basic problem is that we encounter
>>> daily an issue where the mcollective client is unable to communicate with
>>> or find any of the servers. The problem seems to be most clearly seen when
>>> either ActiveMQ is restarted or the MCollective servers are restarted
>>> en-masse like with the mcollectived restart that is part of the logrotate
>>> config. Performing "mco inventory --list-collectives" will result in 0
>>> hosts found. The only way to fix it seems to be to stop all ActiveMQ
>>> instances and then start them one by one.
>>> We are busy preparing to migrate to Choria and Nats but if you have any
>>> advice that would allow us to stabilise the cluster enough to buy us the
>>> time we need to migrate, that would be greatly appreciated.
>>> Our setup is as follows: we have a central hub server with 4 leaf nodes.
>>> The configuration for ActiveMQ has been taken from
>>> The only changes we've made to that config is for the authorization entries
>>> for our users and subcollectives. We've also changed the systemUsage
>>> settings increasing memoryUsage to 512mb, storeUsage to 4gb and tempUsage
>>> to 512mb.
>>> We upgraded ActiveMQ from 5.8 to 5.14.5 last week in the hopes of
>>> increasing the stability but this has not helped. We are running puppet 4
>>> with the puppet-agent 1.10.0 package.
>>> We have the following sysctl settings, which were adapted from the
>>> "Learning MCollective" book (if my memory serves me correctly but they were
>>> added a long time ago):
>>> net.ipv4.tcp_fin_timeout: 15
>>> net.ipv4.tcp_tw_reuse: 1
>>> net.ipv4.tcp_keepalive_time: 300
>>> net.ipv4.tcp_keepalive_intvl: 30
>>> net.ipv4.tcp_keepalive_probes: 5
>>> net.core.somaxconn: 2000
>>> net.core.rmem_default: 256960
>>> net.core.wmem_default: 256960
>>> net.ipv4.tcp_timestamps: 0
>>> We have about 3100 MCollective servers in total. 700 or so connect to
>>> each of the leaf nodes. We also have 14 subcollectives in total matching
>>> the environments and datacenters that we have.
>>> When we have this issue the MCollective logs indicate that they are
>>> connected to ActiveMQ and the ActiveMQ leaf nodes are connected to the
>>> central hub.
>>> Do you have any advice that might help?
You received this message because you are subscribed to the Google Groups
To unsubscribe from this group and stop receiving emails from it, send an email
For more options, visit https://groups.google.com/d/optout.