Thanks. Yes, we do have schedulePeriodForDestinationPurge set. 

On Thursday, 18 May 2017 13:50:11 UTC+1, Charlie Sharpsteen wrote:
>
> In addition to the policyEntries, you'll also need to set 
> schedulePeriodForDestinationPurge on the top-level <broker> element --- 
> otherwise no GC sweeps will occur to clean the inactive queues out. For 
> example:
>
>     <broker brokerName="pe-201645-master.puppetdebug.vlan" xmlns="
> http://activemq.apache.org/schema/core"; persistent="false" 
> schedulePeriodForDestinationPurge="300000">
>
> If it is working, you'll see messages similar to the following in the 
> activemq logs:
>
>     2017-05-04 22:05:48,850 | INFO  | 
> mcollective.reply.pe-201645-master.puppetdebug.vlan_10076.1 Inactive for 
> longer than 300000 ms - removing ... | 
> org.apache.activemq.broker.region.RegionBroker | ActiveMQ 
> Broker[pe-20164nightly-master.puppetdebug.vlan] Scheduler
>
> More info:
>
>   http://activemq.apache.org/delete-inactive-destinations.html
>
> -Charlie
>
> On Wednesday, May 17, 2017 at 9:15:06 PM UTC-7, Richard Houck wrote:
>>
>> Hello,
>>
>> Do you have  Reply Queue Pruning enabled?
>>
>> I had the same problem until I configured this. Hope this helps.
>>
>> MCollective sends replies on uniquely-named, single-use queues with 
>> names like mcollective.reply.<UNIQUE ID>. These have to be deleted after 
>> about five minutes, lest they clog up ActiveMQ’s available memory. By 
>> default, queues live forever, so you have to configure this.
>>
>> Use a <policyEntry> element for *.reply.> queues, with 
>> gcInactiveDestinations set to true and inactiveTimoutBeforeGC set to 
>> 300000 ms (five minutes).
>>
>> <policyEntry queue="*.reply.>" gcInactiveDestinations="true" 
>> inactiveTimoutBeforeGC="300000" />
>>
>>  
>>
>> Try the folloiwng:
>>
>> <policyEntry queue=">" producerFlowControl="true" memoryLimit="20mb" 
>> gcInactiveDestinations="true" inactiveTimoutBeforeGC="10000">.
>>
>>  
>>
>> Desired settings:
>>
>> <policyEntry queue="*.reply.>" producerFlowControl="true" 
>> memoryLimit="20mb" gcInactiveDestinations="true" 
>> inactiveTimoutBeforeGC="300000">
>>
>>
>>
>> Rich
>>
>>
>> On Wednesday, May 17, 2017 at 7:09:32 AM UTC-7, Steven Meunier wrote:
>>>
>>> Hello all
>>>
>>> We're using MCollective with ActiveMQ but are running into problems with 
>>> stability of the ActiveMQ cluster. The basic problem is that we encounter 
>>> daily an issue where the mcollective client is unable to communicate with 
>>> or find any of the servers. The problem seems to be most clearly seen when 
>>> either ActiveMQ is restarted or the MCollective servers are restarted 
>>> en-masse like with the mcollectived restart that is part of the logrotate 
>>> config. Performing "mco inventory --list-collectives" will result in 0 
>>> hosts found. The only way to fix it seems to be to stop all ActiveMQ 
>>> instances and then start them one by one.
>>>
>>> We are busy preparing to migrate to Choria and Nats but if you have any 
>>> advice that would allow us to stabilise the cluster enough to buy us the 
>>> time we need to migrate, that would be greatly appreciated.
>>>
>>> Our setup is as follows: we have a central hub server with 4 leaf nodes. 
>>> The configuration for ActiveMQ has been taken from 
>>> https://github.com/puppetlabs/marionette-collective/tree/master/ext/activemq/examples/multi-broker.
>>>  
>>> The only changes we've made to that config is for the authorization entries 
>>> for our users and subcollectives. We've also changed the systemUsage 
>>> settings increasing memoryUsage to 512mb, storeUsage to 4gb and tempUsage 
>>> to 512mb.
>>>
>>> We upgraded ActiveMQ from 5.8 to 5.14.5 last week in the hopes of 
>>> increasing the stability but this has not helped. We are running puppet 4 
>>> with the puppet-agent 1.10.0 package.
>>>
>>> We have the following sysctl settings, which were adapted from the 
>>> "Learning MCollective" book (if my memory serves me correctly but they were 
>>> added a long time ago):
>>> net.ipv4.tcp_fin_timeout: 15
>>> net.ipv4.tcp_tw_reuse: 1
>>> net.ipv4.tcp_keepalive_time: 300
>>> net.ipv4.tcp_keepalive_intvl: 30
>>> net.ipv4.tcp_keepalive_probes: 5
>>> net.core.somaxconn: 2000
>>> net.core.rmem_default: 256960
>>> net.core.wmem_default: 256960
>>> net.ipv4.tcp_timestamps: 0
>>>
>>> We have about 3100 MCollective servers in total. 700 or so connect to 
>>> each of the leaf nodes. We also have 14 subcollectives in total matching 
>>> the environments and datacenters that we have.
>>>
>>> When we have this issue the MCollective logs indicate that they are 
>>> connected to ActiveMQ and the ActiveMQ leaf nodes are connected to the 
>>> central hub. 
>>>
>>> Do you have any advice that might help?
>>>
>>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"mcollective-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mcollective-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to