I don't have any issues with any specific site until I do mco query for multiple nodes using --nodes.
when I do inventory am getting below, Collective Nodes ========== ===== australia_mcollective 1950 london_mcollective 4120 europe_mcollective 1615 asia_mcollective 3456 china_mcollective 3794 japan_mcollective 7581 america_mcollective 12532 mcollective 35048 inventory details after I run mco query against 1000+ servers list using --nodes and out of that 200 servers not responded, Collective Nodes ========== ===== australia_mcollective 856 london_mcollective 2456 europe_mcollective 1615 asia_mcollective 2678 china_mcollective 1346 japan_mcollective 4156 america_mcollective 7653 mcollective 20760 but if I do mco query for just particular servers using -I then not getting any issues and even when I run for 1000+ nodes using --nodes and if am getting response from all then also I dont have any issues. When some amount of servers not responding for --nodes then only am getting such issues and if I leave it for 5-10 mins then everything coming back to normal. yes, am using sub collectives and each site I have load balancing with multiple activemq brokers. maximum 2000 servers will connect for each broker on every site. Am suspecting that I need to perform some tuning queues but I tried many things which didn't help me. Regards Ravi On Saturday, March 4, 2017 at 12:19:57 PM UTC+5:30, R.I.Pienaar wrote: > > Ok. So do you find that from central to say Japan you have lots of issues? > How is it in Japan from a mco client there communicating with just Japanese > nodes. > > Are you aiming to build one giant 40k node mcollective - not a good idea - > or several smaller ones? Or using sub collectives? > > For sure on many levels a single giant 40k node will not work but with > what you show a number of subcollectives etc should work if it's a hub and > spoke design > > How many nodes per country? And just one activemq there? > > On 4 Mar 2017, at 07:39, [email protected] <javascript:> wrote: > > Sorry yes mco rpc rpcutil ping and I mentioned wrongly. > > you can find my setup on > https://awwapp.com/s/f0123542-5072-43dc-94a1-72f33fbbfcb1/ > > and here is my central server activemq.xml file. Just given you limited > number of sites only here and I have many number of sites and total number > of servers are 40000+ if you combine all sites. Also am using activemq > version 5.14.3. > > > ================================================================================= > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd > http://activemq.apache.org/camel/schema/spring > http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> > > <!-- Allows us to use system properties as variables in this > configuration file --> > <bean > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="locations"> > <value>file:${activemq.base}/conf/credentials.properties</value> > </property> > </bean> > > <!-- > The <broker> element is used to configure the ActiveMQ broker. > --> > > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="central" networkConnectorStartAsync="true" > dataDirectory="${activemq.base}/data" > schedulePeriodForDestinationPurge="60000"> > > <managementContext> > <managementContext createConnector="false"/> > </managementContext> > > <persistenceAdapter> > <kahaDB directory="${activemq.data}/kahadb" > journalMaxFileLength="64mb" preallocationStrategy="zeros" > indexCacheSize="30000" journalDiskSyncStrategy="never" maxAsyncJobs="30000" > indexWriteBatchSize="30000" /> > </persistenceAdapter> > > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue=">" producerFlowControl="false" > gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000"/> > <policyEntry topic=">" producerFlowControl="false"/> > </policyEntries> > </policyMap> > </destinationPolicy> > > <networkConnectors> > > <networkConnector > name="america-topic" > uri="static:(tcp://america:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="america_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="america-queue" > uri="static:(tcp://america:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="america_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="canada-topic" > uri="static:(tcp://canada:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="canada_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="canada-queue" > uri="static:(tcp://canada:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="canada_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="london-topic" > uri="static:(tcp://london:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="london_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="london-queue" > uri="static:(tcp://london:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="london_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="australia-topics" > uri="static:(tcp://australia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="australia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="australia-queue" > uri="static:(tcp://australia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="australia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="europe-topic" > uri="static:(tcp://europe:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="europe_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="europe-queue" > uri="static:(tcp://europe:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="europe_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="china-topic" > uri="static:(tcp://china:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="china_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="china-queue" > uri="static:(tcp://china:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="china_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="asia-topic" > uri="static:(tcp://asia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="asia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="asia-queue" > uri="static:(tcp://asia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="asia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="japan-topic" > uri="static:(tcp://japan:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="japan_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="japan-queue" > uri="static:(tcp://japan:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="japan_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > </networkConnectors> > > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage percentOfJvmHeap="70" /> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb"/> > </storeUsage> > <tempUsage> > <tempUsage limit="1 gb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > <transportConnectors> > <transportConnector name="openwire" uri="tcp://0.0.0.0:61616" > updateClusterClients="true"/> > <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61614"/> > </transportConnectors> > > </broker> > > </beans> > ====================================================================== > > Here is one of the site's activemq.xml file which identical on all sites, > > ====================================================================== > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd > http://activemq.apache.org/camel/schema/spring > http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> > > <!-- Allows us to use system properties as variables in this > configuration file --> > <bean > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="locations"> > <value>file:${activemq.base}/conf/credentials.properties</value> > </property> > </bean> > > <!-- > The <broker> element is used to configure the ActiveMQ broker. > --> > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="america" dataDirectory="${activemq.base}/data" > networkConnectorStartAsync="true" schedulePeriodForDestinationPurge="60000"> > > <managementContext> > <managementContext createConnector="false"/> > </managementContext> > > <persistenceAdapter> > <kahaDB directory="${activemq.data}/kahadb"/> > </persistenceAdapter> > > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue=">" producerFlowControl="false" > gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" /> > <policyEntry topic=">" producerFlowControl="false"/> > </policyEntries> > </policyMap> > </destinationPolicy> > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage percentOfJvmHeap="70"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb"/> > </storeUsage> > <tempUsage> > <tempUsage limit="100 mb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > > <transportConnectors> > <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/> > <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/> > </transportConnectors> > > </broker> > > </beans> > ======================================================================== > > here is the wrapper.conf which is identical for all sites included central > one, > > ======================================================================== > # ------------------------------------------------------------------------ > # Licensed to the Apache Software Foundation (ASF) under one or more > # contributor license agreements. See the NOTICE file distributed with > # this work for additional information regarding copyright ownership. > # The ASF licenses this file to You under the Apache License, Version 2.0 > # (the "License"); you may not use this file except in compliance with > # the License. You may obtain a copy of the License at > # > # http://www.apache.org/licenses/LICENSE-2.0 > # > # Unless required by applicable law or agreed to in writing, software > # distributed under the License is distributed on an "AS IS" BASIS, > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > # See the License for the specific language governing permissions and > # limitations under the License. > # ------------------------------------------------------------------------ > > #******************************************************************** > # Wrapper Properties > #******************************************************************** > > #wrapper.debug=TRUE > set.default.ACTIVEMQ_HOME=/usr/share/activemq > set.default.ACTIVEMQ_BASE=/usr/share/activemq > set.default.ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf > set.default.ACTIVEMQ_DATA=%ACTIVEMQ_BASE%/data > wrapper.working.dir=/ > > # Java Application > wrapper.java.command=java > > # Java Main class. This class must implement the WrapperListener interface > # or guarantee that the WrapperManager class is initialized. Helper > # classes are provided to do this for you. See the Integration section > # of the documentation for details. > wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp > > # Java Classpath (include wrapper.jar) Add class path elements as > # needed starting from 1 > wrapper.java.classpath.1=%ACTIVEMQ_HOME%/bin/wrapper.jar > wrapper.java.classpath.2=%ACTIVEMQ_HOME%/bin/activemq.jar > > # Java Library Path (location of Wrapper.DLL or libwrapper.so) > wrapper.java.library.path.1=%ACTIVEMQ_HOME%/bin > > # Java Additional Parameters > # note that n is the parameter number starting from 1. > wrapper.java.additional.1=-Dactivemq.home=%ACTIVEMQ_HOME% > wrapper.java.additional.2=-Dactivemq.base=%ACTIVEMQ_BASE% > wrapper.java.additional.3=-Djavax.net.ssl.keyStorePassword=password > wrapper.java.additional.4=-Djavax.net.ssl.trustStorePassword=password > > wrapper.java.additional.5=-Djavax.net.ssl.keyStore=%ACTIVEMQ_CONF%/broker.ks > > wrapper.java.additional.6=-Djavax.net.ssl.trustStore=%ACTIVEMQ_CONF%/broker.ts > wrapper.java.additional.7=-Dcom.sun.management.jmxremote > > wrapper.java.additional.8=-Dorg.apache.activemq.UseDedicatedTaskRunner=false > > wrapper.java.additional.9=-Djava.util.logging.config.file=logging.properties > wrapper.java.additional.10=-Dactivemq.conf=%ACTIVEMQ_CONF% > wrapper.java.additional.11=-Dactivemq.data=%ACTIVEMQ_DATA% > > wrapper.java.additional.12=-Djava.security.auth.login.config=%ACTIVEMQ_CONF%/login.config > wrapper.java.additional.13=-Dhawtio.realm=activemq > wrapper.java.additional.14=-Dhawtio.role=admins > > wrapper.java.additional.15=-Dhawtio.rolePrincipalClasses=org.apache.activemq.jaas.GroupPrincipal > wrapper.java.additional.16=-Xms16384m > wrapper.java.additional.17=-Xmx16384m > wrapper.java.additional.18=-XX:+UseG1GC > > wrapper.java.additional.19=-Dorg.apache.activemq.kahaDB.files.skipMetadataUpdate=true > wrapper.java.additional.20=-XX:MaxMetaspaceSize=16000m > > # Uncomment to enable jmx > #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.port=1616 > > #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.authenticate=false > #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.ssl=false > > # Uncomment to enable YourKit profiling > #wrapper.java.additional.n=-Xrunyjpagent > > # Uncomment to enable remote debugging > #wrapper.java.additional.n=-Xdebug -Xnoagent -Djava.compiler=NONE > > #wrapper.java.additional.n=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 > > # Initial Java Heap Size (in MB) > #wrapper.java.initmemory=256 > > # Maximum Java Heap Size (in MB) > #wrapper.java.maxmemory=12288 > > # Application parameters. Add parameters as needed starting from 1 > wrapper.app.parameter.1=org.apache.activemq.console.Main > wrapper.app.parameter.2=start > > #******************************************************************** > # Wrapper Logging Properties > #******************************************************************** > # Format of output for the console. (See docs for formats) > wrapper.console.format=PM > > # Log Level for console output. (See docs for log levels) > wrapper.console.loglevel=INFO > > # Log file to use for wrapper output logging. > wrapper.logfile=%ACTIVEMQ_DATA%/wrapper.log > > # Format of output for the log file. (See docs for formats) > wrapper.logfile.format=LPTM > > # Log Level for log file output. (See docs for log levels) > wrapper.logfile.loglevel=INFO > > # Maximum size that the log file will be allowed to grow to before > # the log is rolled. Size is specified in bytes. The default value > # of 0, disables log rolling. May abbreviate with the 'k' (kb) or > # 'm' (mb) suffix. For example: 10m = 10 megabytes. > wrapper.logfile.maxsize=0 > > # Maximum number of rolled log files which will be allowed before old > # files are deleted. The default value of 0 implies no limit. > wrapper.logfile.maxfiles=0 > > # Log Level for sys/event log output. (See docs for log levels) > wrapper.syslog.loglevel=NONE > > #******************************************************************** > # Wrapper Windows Properties > #******************************************************************** > # Title to use when running as a console > wrapper.console.title=ActiveMQ > > #******************************************************************** > # Wrapper Windows NT/2000/XP Service Properties > #******************************************************************** > # WARNING - Do not modify any of these properties when an application > # using this configuration file has been installed as a service. > # Please uninstall the service before modifying this section. The > # service can then be reinstalled. > > # Name of the service > wrapper.ntservice.name=ActiveMQ > > # Display name of the service > wrapper.ntservice.displayname=ActiveMQ > > # Description of the service > wrapper.ntservice.description=ActiveMQ Broker > > # Service dependencies. Add dependencies as needed starting from 1 > wrapper.ntservice.dependency.1= > > # Mode in which the service is installed. AUTO_START or DEMAND_START > wrapper.ntservice.starttype=AUTO_START > > # Allow the service to interact with the desktop. > wrapper.ntservice.interactive=false > > ============================================================================= > > > Regards > Ravi > > > On Saturday, March 4, 2017 at 11:31:49 AM UTC+5:30, R.I.Pienaar wrote: >> >> You'll need to give some details on how activemq is configured >> >> mco ping does not use the --nodes argument. Try mco rpc rpcutil ping. >> That will use it. >> >> --- >> R.I.Pienaar >> >> On 4 Mar 2017, at 05:29, [email protected] wrote: >> >> Hi, >> >> I want to use mcollective on geographically and attempted to configure >> ActiveMQ network of brokers for that. After the configuration everything is >> working fine as expected but only problem is, when I run any mco query for >> multiple servers then its getting disconnected for sometime when many of >> the not responded. >> >> For example, I have connected 1000 clients for each location of ActiveMQ >> broker, and am deploying mcollective on many servers and I have the entire >> list of servers(458) which am deploying it. So when I run mco ping or any >> mco queries with --nodes=<server_list_file> geographically am getting >> response from 243 servers which means mcollective configured successfully >> on these servers and no response from other servers which means mcollective >> not configured yet on those servers. But the problem is here many servers >> are not responded for mcollective and due to this when I run the mco query >> on connected servers am not getting response. But If I leave for few >> minutes and then run the same query against working servers then am getting >> response. Am suspecting that when I run mco query and many servers are not >> responded then middleware takes time to clear pending queues or something >> happening. Any idea to tune this? >> >> >> Regards >> Ravi >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "mcollective-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "mcollective-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
