Hi,

Sorry I didn't get you exactly. Am already having load balancing on all 
remote sites and have only one broker on central. Do you want me to try 
with load balancing for central one also? 

Also I just remember that I have set ttl to 300 to avoid time drift issue 
for some of the servers. Not sure if this is causing the issue and also not 
sure what limit is reached. can you check my config and suggests me if any 
limit I can tune it to overcome this issue?


Regards
Ravi

On Saturday, March 4, 2017 at 1:55:46 PM UTC+5:30, R.I.Pienaar wrote:
>
> ok, and with --nodes it must send individual messages there's no other 
> way 
>
> This would put significant pressure on your broker 2 broker connections, 
> either ActiveMQ is marking some remote brokers as 'slow' and so they 
> must back 
> off and wait before they can get messages again or some other limits are 
> reached. 
>
> you can perhaps try make multiple connection from your remote sites to 
> the 
> hub - even in a single ActiveMQ 2 x broker to broker connections to the 
> same 
> upstream will result in it load balancing across those TCP connections 
>
> Other than that, I am not too sure what we can try - what is the 
> motivation 
> for such huge requests with --nodes ? 
>
> On Sat, Mar 4, 2017, at 09:11, [email protected] <javascript:> wrote: 
> > yeah am also thinking that some optimisations I have to do but I tried 
> > lot 
> > which didn't help. If you would help me with optimization that would be 
> > great. 
> > 
> > I put my client in debug mode and seeing it publishing 1 message per 
> > node, 
> > since am using direct_addressing on both client and server config. 
> > 
> > `publish' Sending a direct message to ActiveMQ target 
> > '/queue/mcollective.nodes' with headers '{"mc_identity"=>"xxxxxxxx", 
> > "timestamp"=>"1488614346000", 
> > "reply-to"=>"/queue/mcollective.reply.yyyyyyyyyy_16102", 
> > "expires"=>"1488614656000"}' 
> > 
> > 
> > Regards 
> > Ravi 
> > 
> > On Saturday, March 4, 2017 at 1:12:29 PM UTC+5:30, R.I.Pienaar wrote: 
> > > 
> > > hey, 
> > > 
> > > OK See 
> > > 
> > > So the technical difference here is that without --nodes mcollective 
> > > publishes a message and relies on the broker to broadcast the messages 
> > > but with --nodes it has to publish individual messages thus putting 
> > > quite the strain on the middleware. 
> > > 
> > > I recall there being some optimisations around when it does and does 
> not 
> > > publish individual messages but if you put your client into debug mode 
> > > you'll either see it publishing 1 message per node in your nodes file 
> or 
> > > something like "Sending a broadcast message to ActiveMQ" 
> > > 
> > > which do you see? 
> > > 
> > > On Sat, Mar 4, 2017, at 08:01, [email protected] <javascript:> 
> wrote: 
> > > > I don't have any issues with any specific site until I do mco query 
> for 
> > > > multiple nodes using --nodes. 
> > > > 
> > > > when I do inventory am getting below, 
> > > > 
> > > > Collective                     Nodes 
> > > >    ==========                     ===== 
> > > >    australia_mcollective               1950 
> > > >    london_mcollective               4120 
> > > >    europe_mcollective                1615 
> > > >    asia_mcollective                3456 
> > > >    china_mcollective               3794 
> > > >    japan_mcollective               7581 
> > > >    america_mcollective                12532 
> > > >    mcollective                    35048 
> > > > 
> > > > inventory details after I run mco query against 1000+ servers list 
> using 
> > > > --nodes and out of that 200 servers not responded, 
> > > > 
> > > > Collective                     Nodes 
> > > >    ==========                     ===== 
> > > >    australia_mcollective               856 
> > > >    london_mcollective               2456 
> > > >    europe_mcollective                1615 
> > > >    asia_mcollective                2678 
> > > >    china_mcollective               1346 
> > > >    japan_mcollective               4156 
> > > >    america_mcollective                7653 
> > > >    mcollective                    20760 
> > > > 
> > > > but if I do mco query for just particular servers using -I then not 
> > > > getting 
> > > > any issues and even when I run for 1000+ nodes using --nodes and if 
> am 
> > > > getting response from all then also I dont have any issues. When 
> some 
> > > > amount of servers not responding for --nodes then only am getting 
> such 
> > > > issues and if I leave it for 5-10 mins then everything coming back 
> to 
> > > > normal. 
> > > > 
> > > > yes, am using sub collectives and each site I have load balancing 
> with 
> > > > multiple activemq brokers. maximum 2000 servers will connect for 
> each 
> > > > broker on every site. 
> > > > 
> > > > Am suspecting that I need to perform some tuning queues but I tried 
> many 
> > > > things which didn't help me. 
> > > > 
> > > > 
> > > > Regards 
> > > > Ravi 
> > > > 
> > > > On Saturday, March 4, 2017 at 12:19:57 PM UTC+5:30, R.I.Pienaar 
> wrote: 
> > > > > 
> > > > > Ok. So do you find that from central to say Japan you have lots of 
> > > issues? 
> > > > > How is it in Japan from a mco client there communicating with just 
> > > Japanese 
> > > > > nodes. 
> > > > > 
> > > > > Are you aiming to build one giant 40k node mcollective - not a 
> good 
> > > idea - 
> > > > > or several smaller ones? Or using sub collectives? 
> > > > > 
> > > > > For sure on many levels a single giant 40k node will not work but 
> with 
> > > > > what you show a number of subcollectives etc should work if it's a 
> hub 
> > > and 
> > > > > spoke design 
> > > > > 
> > > > > How many nodes per country? And just one activemq there? 
> > > > > 
> > > > > On 4 Mar 2017, at 07:39, [email protected] <javascript:> wrote: 
> > > > > 
> > > > > Sorry yes mco rpc rpcutil ping and I mentioned wrongly. 
> > > > > 
> > > > > you can find my setup on 
> > > > > https://awwapp.com/s/f0123542-5072-43dc-94a1-72f33fbbfcb1/ 
> > > > > 
> > > > > and here is my central server activemq.xml file. Just given you 
> > > limited 
> > > > > number of sites only here and I have many number of sites and 
> total 
> > > number 
> > > > > of servers are 40000+ if you combine all sites. Also am using 
> activemq 
> > > > > version 5.14.3. 
> > > > > 
> > > > > 
> > > > > 
> > > 
> =================================================================================
>  
>
> > > 
> > > > > <beans 
> > > > >   xmlns="http://www.springframework.org/schema/beans"; 
> > > > >   xmlns:amq="http://activemq.apache.org/schema/core"; 
> > > > >   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
> > > > >   xsi:schemaLocation="http://www.springframework.org/schema/beans 
> > > > > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd 
> > > > >   http://activemq.apache.org/schema/core 
> > > > > http://activemq.apache.org/schema/core/activemq-core.xsd 
> > > > >   http://activemq.apache.org/camel/schema/spring 
> > > > > http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> 
> > > > > 
> > > > >   <!-- Allows us to use system properties as variables in this 
> > > > > configuration file --> 
> > > > >   <bean 
> > > > > 
> > > 
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>  
>
> > > 
> > > > >     <property name="locations"> 
> > > > >       
> <value>file:${activemq.base}/conf/credentials.properties</value> 
> > > > >     </property> 
> > > > >   </bean> 
> > > > > 
> > > > > <!-- 
> > > > > The <broker> element is used to configure the ActiveMQ broker. 
> > > > > --> 
> > > > > 
> > > > > <broker xmlns="http://activemq.apache.org/schema/core"; 
> > > > > brokerName="central" networkConnectorStartAsync="true" 
> > > > > dataDirectory="${activemq.base}/data" 
> > > > > schedulePeriodForDestinationPurge="60000"> 
> > > > > 
> > > > >   <managementContext> 
> > > > >     <managementContext createConnector="false"/> 
> > > > >   </managementContext> 
> > > > > 
> > > > >   <persistenceAdapter> 
> > > > >     <kahaDB directory="${activemq.data}/kahadb" 
> > > > > journalMaxFileLength="64mb" preallocationStrategy="zeros" 
> > > > > indexCacheSize="30000" journalDiskSyncStrategy="never" 
> > > maxAsyncJobs="30000" 
> > > > > indexWriteBatchSize="30000" /> 
> > > > >   </persistenceAdapter> 
> > > > > 
> > > > >   <destinationPolicy> 
> > > > >     <policyMap> 
> > > > >       <policyEntries> 
> > > > >         <policyEntry queue=">" producerFlowControl="false" 
> > > > > gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000"/> 
> > > > >         <policyEntry topic=">" producerFlowControl="false"/> 
> > > > >       </policyEntries> 
> > > > >     </policyMap> 
> > > > >   </destinationPolicy> 
> > > > > 
> > > > >   <networkConnectors> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="america-topic" 
> > > > >       uri="static:(tcp://america:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="america_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="america-queue" 
> > > > >       uri="static:(tcp://america:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="america_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="canada-topic" 
> > > > >       uri="static:(tcp://canada:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="canada_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="canada-queue" 
> > > > >       uri="static:(tcp://canada:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="canada_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="london-topic" 
> > > > >       uri="static:(tcp://london:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="london_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="london-queue" 
> > > > >       uri="static:(tcp://london:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="london_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="australia-topics" 
> > > > >       uri="static:(tcp://australia:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="australia_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="australia-queue" 
> > > > >       uri="static:(tcp://australia:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="australia_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="europe-topic" 
> > > > >       uri="static:(tcp://europe:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="europe_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="europe-queue" 
> > > > >       uri="static:(tcp://europe:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="europe_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="china-topic" 
> > > > >       uri="static:(tcp://china:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="china_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="china-queue" 
> > > > >       uri="static:(tcp://china:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="china_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="asia-topic" 
> > > > >       uri="static:(tcp://asia:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="asia_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="asia-queue" 
> > > > >       uri="static:(tcp://asia:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="asia_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="japan-topic" 
> > > > >       uri="static:(tcp://japan:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >       dynamicOnly="true"> 
> > > > >       <excludedDestinations> 
> > > > >         <queue physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <topic physicalName="ActiveMQ.>" /> 
> > > > >         <topic physicalName="mcollective.>" /> 
> > > > >         <topic physicalName="japan_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >     <networkConnector 
> > > > >       name="japan-queue" 
> > > > >       uri="static:(tcp://japan:61616)" 
> > > > >       duplex="true" 
> > > > >           decreaseNetworkConsumerPriority="true" 
> > > > >           prefetchSize="20000" 
> > > > >       dynamicOnly="true" 
> > > > >       conduitSubscriptions="false"> 
> > > > >       <excludedDestinations> 
> > > > >         <topic physicalName=">" /> 
> > > > >       </excludedDestinations> 
> > > > >       <dynamicallyIncludedDestinations> 
> > > > >         <queue physicalName="mcollective.>" /> 
> > > > >         <queue physicalName="japan_mcollective.>" /> 
> > > > >       </dynamicallyIncludedDestinations> 
> > > > >     </networkConnector> 
> > > > > 
> > > > >   </networkConnectors> 
> > > > > 
> > > > > 
> > > > >   <systemUsage> 
> > > > >       <systemUsage> 
> > > > >           <memoryUsage> 
> > > > >               <memoryUsage percentOfJvmHeap="70" /> 
> > > > >           </memoryUsage> 
> > > > >           <storeUsage> 
> > > > >               <storeUsage limit="1 gb"/> 
> > > > >           </storeUsage> 
> > > > >           <tempUsage> 
> > > > >               <tempUsage limit="1 gb"/> 
> > > > >           </tempUsage> 
> > > > >       </systemUsage> 
> > > > >   </systemUsage> 
> > > > > 
> > > > >   <transportConnectors> 
> > > > >     <transportConnector name="openwire" uri="tcp://0.0.0.0:61616" 
> > > > > updateClusterClients="true"/> 
> > > > >     <transportConnector name="stomp+nio" uri="stomp+nio:// 
> > > 0.0.0.0:61614"/> 
> > > > >   </transportConnectors> 
> > > > > 
> > > > > </broker> 
> > > > > 
> > > > > </beans> 
> > > > > 
> ====================================================================== 
> > > > > 
> > > > > Here is one of the site's activemq.xml file which identical on all 
> > > sites, 
> > > > > 
> > > > > 
> ====================================================================== 
> > > > > <beans 
> > > > >   xmlns="http://www.springframework.org/schema/beans"; 
> > > > >   xmlns:amq="http://activemq.apache.org/schema/core"; 
> > > > >   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
> > > > >   xsi:schemaLocation="http://www.springframework.org/schema/beans 
> > > > > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd 
> > > > >   http://activemq.apache.org/schema/core 
> > > > > http://activemq.apache.org/schema/core/activemq-core.xsd 
> > > > >   http://activemq.apache.org/camel/schema/spring 
> > > > > http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> 
> > > > > 
> > > > >   <!-- Allows us to use system properties as variables in this 
> > > > > configuration file --> 
> > > > >   <bean 
> > > > > 
> > > 
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>  
>
> > > 
> > > > >     <property name="locations"> 
> > > > >       
> <value>file:${activemq.base}/conf/credentials.properties</value> 
> > > > >     </property> 
> > > > >   </bean> 
> > > > > 
> > > > > <!-- 
> > > > > The <broker> element is used to configure the ActiveMQ broker. 
> > > > > --> 
> > > > > <broker xmlns="http://activemq.apache.org/schema/core"; 
> > > > > brokerName="america" dataDirectory="${activemq.base}/data" 
> > > > > networkConnectorStartAsync="true" 
> > > schedulePeriodForDestinationPurge="60000"> 
> > > > > 
> > > > >   <managementContext> 
> > > > >     <managementContext createConnector="false"/> 
> > > > >   </managementContext> 
> > > > > 
> > > > >   <persistenceAdapter> 
> > > > >     <kahaDB directory="${activemq.data}/kahadb"/> 
> > > > >   </persistenceAdapter> 
> > > > > 
> > > > >   <destinationPolicy> 
> > > > >     <policyMap> 
> > > > >       <policyEntries> 
> > > > >         <policyEntry queue=">" producerFlowControl="false" 
> > > > > gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" /> 
> > > > >         <policyEntry topic=">" producerFlowControl="false"/> 
> > > > >       </policyEntries> 
> > > > >     </policyMap> 
> > > > >   </destinationPolicy> 
> > > > > 
> > > > >   <systemUsage> 
> > > > >     <systemUsage> 
> > > > >       <memoryUsage> 
> > > > >         <memoryUsage percentOfJvmHeap="70"/> 
> > > > >       </memoryUsage> 
> > > > >       <storeUsage> 
> > > > >         <storeUsage limit="1 gb"/> 
> > > > >       </storeUsage> 
> > > > >       <tempUsage> 
> > > > >         <tempUsage limit="100 mb"/> 
> > > > >       </tempUsage> 
> > > > >     </systemUsage> 
> > > > >   </systemUsage> 
> > > > > 
> > > > > 
> > > > >   <transportConnectors> 
> > > > >     <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/> 
>
> > > > >     <transportConnector name="stomp+nio" uri="stomp+nio:// 
> > > 0.0.0.0:61613"/> 
> > > > >   </transportConnectors> 
> > > > > 
> > > > > </broker> 
> > > > > 
> > > > > </beans> 
> > > > > 
> > > 
> ======================================================================== 
> > > > > 
> > > > > here is the wrapper.conf which is identical for all sites included 
> > > central 
> > > > > one, 
> > > > > 
> > > > > 
> > > 
> ======================================================================== 
> > > > > # 
> > > 
> ------------------------------------------------------------------------ 
> > > > > # Licensed to the Apache Software Foundation (ASF) under one or 
> more 
> > > > > # contributor license agreements.  See the NOTICE file distributed 
> > > with 
> > > > > # this work for additional information regarding copyright 
> ownership. 
> > > > > # The ASF licenses this file to You under the Apache License, 
> Version 
> > > 2.0 
> > > > > # (the "License"); you may not use this file except in compliance 
> with 
> > > > > # the License.  You may obtain a copy of the License at 
> > > > > # 
> > > > > # http://www.apache.org/licenses/LICENSE-2.0 
> > > > > # 
> > > > > # Unless required by applicable law or agreed to in writing, 
> software 
> > > > > # distributed under the License is distributed on an "AS IS" 
> BASIS, 
> > > > > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
> > > implied. 
> > > > > # See the License for the specific language governing permissions 
> and 
> > > > > # limitations under the License. 
> > > > > # 
> > > 
> ------------------------------------------------------------------------ 
> > > > > 
> > > > > 
> #******************************************************************** 
> > > > > # Wrapper Properties 
> > > > > 
> #******************************************************************** 
> > > > > 
> > > > > #wrapper.debug=TRUE 
> > > > > set.default.ACTIVEMQ_HOME=/usr/share/activemq 
> > > > > set.default.ACTIVEMQ_BASE=/usr/share/activemq 
> > > > > set.default.ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf 
> > > > > set.default.ACTIVEMQ_DATA=%ACTIVEMQ_BASE%/data 
> > > > > wrapper.working.dir=/ 
> > > > > 
> > > > > # Java Application 
> > > > > wrapper.java.command=java 
> > > > > 
> > > > > # Java Main class.  This class must implement the WrapperListener 
> > > interface 
> > > > > #  or guarantee that the WrapperManager class is initialized. 
>  Helper 
> > > > > #  classes are provided to do this for you.  See the Integration 
> > > section 
> > > > > #  of the documentation for details. 
> > > > > wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp 
> > > > > 
> > > > > # Java Classpath (include wrapper.jar)  Add class path elements as 
> > > > > #  needed starting from 1 
> > > > > wrapper.java.classpath.1=%ACTIVEMQ_HOME%/bin/wrapper.jar 
> > > > > wrapper.java.classpath.2=%ACTIVEMQ_HOME%/bin/activemq.jar 
> > > > > 
> > > > > # Java Library Path (location of Wrapper.DLL or libwrapper.so) 
> > > > > wrapper.java.library.path.1=%ACTIVEMQ_HOME%/bin 
> > > > > 
> > > > > # Java Additional Parameters 
> > > > > # note that n is the parameter number starting from 1. 
> > > > > wrapper.java.additional.1=-Dactivemq.home=%ACTIVEMQ_HOME% 
> > > > > wrapper.java.additional.2=-Dactivemq.base=%ACTIVEMQ_BASE% 
> > > > > 
> wrapper.java.additional.3=-Djavax.net.ssl.keyStorePassword=password 
> > > > > 
> wrapper.java.additional.4=-Djavax.net.ssl.trustStorePassword=password 
> > > > > 
> > > > > 
> > > 
> wrapper.java.additional.5=-Djavax.net.ssl.keyStore=%ACTIVEMQ_CONF%/broker.ks 
>
> > > 
> > > > > 
> > > > > 
> > > 
> wrapper.java.additional.6=-Djavax.net.ssl.trustStore=%ACTIVEMQ_CONF%/broker.ts
>  
>
> > > 
> > > > > wrapper.java.additional.7=-Dcom.sun.management.jmxremote 
> > > > > 
> > > > > 
> > > 
> wrapper.java.additional.8=-Dorg.apache.activemq.UseDedicatedTaskRunner=false 
>
> > > 
> > > > > 
> > > > > 
> > > 
> wrapper.java.additional.9=-Djava.util.logging.config.file=logging.properties 
>
> > > 
> > > > > wrapper.java.additional.10=-Dactivemq.conf=%ACTIVEMQ_CONF% 
> > > > > wrapper.java.additional.11=-Dactivemq.data=%ACTIVEMQ_DATA% 
> > > > > 
> > > > > 
> > > 
> wrapper.java.additional.12=-Djava.security.auth.login.config=%ACTIVEMQ_CONF%/login.config
>  
>
> > > 
> > > > > wrapper.java.additional.13=-Dhawtio.realm=activemq 
> > > > > wrapper.java.additional.14=-Dhawtio.role=admins 
> > > > > 
> > > > > 
> > > 
> wrapper.java.additional.15=-Dhawtio.rolePrincipalClasses=org.apache.activemq.jaas.GroupPrincipal
>  
>
> > > 
> > > > > wrapper.java.additional.16=-Xms16384m 
> > > > > wrapper.java.additional.17=-Xmx16384m 
> > > > > wrapper.java.additional.18=-XX:+UseG1GC 
> > > > > 
> > > > > 
> > > 
> wrapper.java.additional.19=-Dorg.apache.activemq.kahaDB.files.skipMetadataUpdate=true
>  
>
> > > 
> > > > > wrapper.java.additional.20=-XX:MaxMetaspaceSize=16000m 
> > > > > 
> > > > > # Uncomment to enable jmx 
> > > > > 
> #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.port=1616 
> > > > > 
> > > > > 
> > > 
> #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.authenticate=false 
>
> > > 
> > > > > 
> #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.ssl=false 
> > > > > 
> > > > > # Uncomment to enable YourKit profiling 
> > > > > #wrapper.java.additional.n=-Xrunyjpagent 
> > > > > 
> > > > > # Uncomment to enable remote debugging 
> > > > > #wrapper.java.additional.n=-Xdebug -Xnoagent -Djava.compiler=NONE 
> > > > > 
> > > > > 
> > > 
> #wrapper.java.additional.n=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005
>  
>
> > > 
> > > > > 
> > > > > # Initial Java Heap Size (in MB) 
> > > > > #wrapper.java.initmemory=256 
> > > > > 
> > > > > # Maximum Java Heap Size (in MB) 
> > > > > #wrapper.java.maxmemory=12288 
> > > > > 
> > > > > # Application parameters.  Add parameters as needed starting from 
> 1 
> > > > > wrapper.app.parameter.1=org.apache.activemq.console.Main 
> > > > > wrapper.app.parameter.2=start 
> > > > > 
> > > > > 
> #******************************************************************** 
> > > > > # Wrapper Logging Properties 
> > > > > 
> #******************************************************************** 
> > > > > # Format of output for the console.  (See docs for formats) 
> > > > > wrapper.console.format=PM 
> > > > > 
> > > > > # Log Level for console output.  (See docs for log levels) 
> > > > > wrapper.console.loglevel=INFO 
> > > > > 
> > > > > # Log file to use for wrapper output logging. 
> > > > > wrapper.logfile=%ACTIVEMQ_DATA%/wrapper.log 
> > > > > 
> > > > > # Format of output for the log file.  (See docs for formats) 
> > > > > wrapper.logfile.format=LPTM 
> > > > > 
> > > > > # Log Level for log file output.  (See docs for log levels) 
> > > > > wrapper.logfile.loglevel=INFO 
> > > > > 
> > > > > # Maximum size that the log file will be allowed to grow to before 
> > > > > #  the log is rolled. Size is specified in bytes.  The default 
> value 
> > > > > #  of 0, disables log rolling.  May abbreviate with the 'k' (kb) 
> or 
> > > > > #  'm' (mb) suffix.  For example: 10m = 10 megabytes. 
> > > > > wrapper.logfile.maxsize=0 
> > > > > 
> > > > > # Maximum number of rolled log files which will be allowed before 
> old 
> > > > > #  files are deleted.  The default value of 0 implies no limit. 
> > > > > wrapper.logfile.maxfiles=0 
> > > > > 
> > > > > # Log Level for sys/event log output.  (See docs for log levels) 
> > > > > wrapper.syslog.loglevel=NONE 
> > > > > 
> > > > > 
> #******************************************************************** 
> > > > > # Wrapper Windows Properties 
> > > > > 
> #******************************************************************** 
> > > > > # Title to use when running as a console 
> > > > > wrapper.console.title=ActiveMQ 
> > > > > 
> > > > > 
> #******************************************************************** 
> > > > > # Wrapper Windows NT/2000/XP Service Properties 
> > > > > 
> #******************************************************************** 
> > > > > # WARNING - Do not modify any of these properties when an 
> application 
> > > > > #  using this configuration file has been installed as a service. 
> > > > > #  Please uninstall the service before modifying this section. 
>  The 
> > > > > #  service can then be reinstalled. 
> > > > > 
> > > > > # Name of the service 
> > > > > wrapper.ntservice.name=ActiveMQ 
> > > > > 
> > > > > # Display name of the service 
> > > > > wrapper.ntservice.displayname=ActiveMQ 
> > > > > 
> > > > > # Description of the service 
> > > > > wrapper.ntservice.description=ActiveMQ Broker 
> > > > > 
> > > > > # Service dependencies.  Add dependencies as needed starting from 
> 1 
> > > > > wrapper.ntservice.dependency.1= 
> > > > > 
> > > > > # Mode in which the service is installed.  AUTO_START or 
> DEMAND_START 
> > > > > wrapper.ntservice.starttype=AUTO_START 
> > > > > 
> > > > > # Allow the service to interact with the desktop. 
> > > > > wrapper.ntservice.interactive=false 
> > > > > 
> > > > > 
> > > 
> ============================================================================= 
>
> > > 
> > > > > 
> > > > > 
> > > > > Regards 
> > > > > Ravi 
> > > > > 
> > > > > 
> > > > > On Saturday, March 4, 2017 at 11:31:49 AM UTC+5:30, R.I.Pienaar 
> wrote: 
> > > > >> 
> > > > >> You'll need to give some details on how activemq is configured 
> > > > >> 
> > > > >> mco ping does not use the --nodes argument. Try  mco rpc rpcutil 
> > > ping. 
> > > > >> That will use it. 
> > > > >> 
> > > > >> --- 
> > > > >> R.I.Pienaar 
> > > > >> 
> > > > >> On 4 Mar 2017, at 05:29, [email protected] wrote: 
> > > > >> 
> > > > >> Hi, 
> > > > >> 
> > > > >> I want to use mcollective on geographically and attempted to 
> > > configure 
> > > > >> ActiveMQ network of brokers for that. After the configuration 
> > > everything is 
> > > > >> working fine as expected but only problem is, when I run any mco 
> > > query for 
> > > > >> multiple servers then its getting disconnected for sometime when 
> many 
> > > of 
> > > > >> the not responded. 
> > > > >> 
> > > > >> For example, I have connected 1000 clients for each location of 
> > > ActiveMQ 
> > > > >> broker, and am deploying mcollective on many servers and I have 
> the 
> > > entire 
> > > > >> list of servers(458) which am deploying it. So when I run mco 
> ping or 
> > > any 
> > > > >> mco queries with --nodes=<server_list_file> geographically am 
> getting 
> > > > >> response from 243 servers which means mcollective configured 
> > > successfully 
> > > > >> on these servers and no response from other servers which means 
> > > mcollective 
> > > > >> not configured yet on those servers. But the problem is here many 
> > > servers 
> > > > >> are not responded for mcollective and due to this when I run the 
> mco 
> > > query 
> > > > >> on connected servers am not getting response. But If I leave for 
> few 
> > > > >> minutes and then run the same query against working servers then 
> am 
> > > getting 
> > > > >> response. Am suspecting that when I run mco query and many 
> servers 
> > > are not 
> > > > >> responded then middleware takes time to clear pending queues or 
> > > something 
> > > > >> happening. Any idea to tune this? 
> > > > >> 
> > > > >> 
> > > > >> Regards 
> > > > >> Ravi 
> > > > >> 
> > > > >> -- 
> > > > >> 
> > > > >> --- 
> > > > >> You received this message because you are subscribed to the 
> Google 
> > > Groups 
> > > > >> "mcollective-users" group. 
> > > > >> To unsubscribe from this group and stop receiving emails from it, 
> > > send an 
> > > > >> email to [email protected]. 
> > > > >> For more options, visit https://groups.google.com/d/optout. 
> > > > >> 
> > > > >> -- 
> > > > > 
> > > > > --- 
> > > > > You received this message because you are subscribed to the Google 
> > > Groups 
> > > > > "mcollective-users" group. 
> > > > > To unsubscribe from this group and stop receiving emails from it, 
> send 
> > > an 
> > > > > email to [email protected] <javascript:>. 
> > > > > For more options, visit https://groups.google.com/d/optout. 
> > > > > 
> > > > > 
> > > > 
> > > > -- 
> > > > 
> > > > --- 
> > > > You received this message because you are subscribed to the Google 
> > > Groups 
> > > > "mcollective-users" group. 
> > > > To unsubscribe from this group and stop receiving emails from it, 
> send 
> > > an 
> > > > email to [email protected] <javascript:>. 
> > > > For more options, visit https://groups.google.com/d/optout. 
> > > 
> > > 
> > > -- 
> > > R.I.Pienaar / www.devco.net / @ripienaar 
> > > 
> > 
> > -- 
> > 
> > --- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "mcollective-users" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to [email protected] <javascript:>. 
> > For more options, visit https://groups.google.com/d/optout. 
>
>
> -- 
> R.I.Pienaar / www.devco.net / @ripienaar 
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"mcollective-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
For more options, visit https://groups.google.com/d/optout.

Reply via email to