Ok. So do you find that from central to say Japan you have lots of issues? How is it in Japan from a mco client there communicating with just Japanese nodes.
Are you aiming to build one giant 40k node mcollective - not a good idea - or several smaller ones? Or using sub collectives? For sure on many levels a single giant 40k node will not work but with what you show a number of subcollectives etc should work if it's a hub and spoke design How many nodes per country? And just one activemq there? > On 4 Mar 2017, at 07:39, [email protected] wrote: > > Sorry yes mco rpc rpcutil ping and I mentioned wrongly. > > you can find my setup on > https://awwapp.com/s/f0123542-5072-43dc-94a1-72f33fbbfcb1/ > > and here is my central server activemq.xml file. Just given you limited > number of sites only here and I have many number of sites and total number of > servers are 40000+ if you combine all sites. Also am using activemq version > 5.14.3. > > ================================================================================= > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd > http://activemq.apache.org/camel/schema/spring > http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> > > <!-- Allows us to use system properties as variables in this configuration > file --> > <bean > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="locations"> > <value>file:${activemq.base}/conf/credentials.properties</value> > </property> > </bean> > > <!-- > The <broker> element is used to configure the ActiveMQ broker. > --> > > <broker xmlns="http://activemq.apache.org/schema/core"; brokerName="central" > networkConnectorStartAsync="true" dataDirectory="${activemq.base}/data" > schedulePeriodForDestinationPurge="60000"> > > <managementContext> > <managementContext createConnector="false"/> > </managementContext> > > <persistenceAdapter> > <kahaDB directory="${activemq.data}/kahadb" journalMaxFileLength="64mb" > preallocationStrategy="zeros" indexCacheSize="30000" > journalDiskSyncStrategy="never" maxAsyncJobs="30000" > indexWriteBatchSize="30000" /> > </persistenceAdapter> > > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue=">" producerFlowControl="false" > gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000"/> > <policyEntry topic=">" producerFlowControl="false"/> > </policyEntries> > </policyMap> > </destinationPolicy> > > <networkConnectors> > > <networkConnector > name="america-topic" > uri="static:(tcp://america:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="america_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="america-queue" > uri="static:(tcp://america:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="america_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="canada-topic" > uri="static:(tcp://canada:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="canada_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="canada-queue" > uri="static:(tcp://canada:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="canada_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="london-topic" > uri="static:(tcp://london:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="london_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="london-queue" > uri="static:(tcp://london:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="london_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="australia-topics" > uri="static:(tcp://australia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="australia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="australia-queue" > uri="static:(tcp://australia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="australia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="europe-topic" > uri="static:(tcp://europe:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="europe_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="europe-queue" > uri="static:(tcp://europe:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="europe_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="china-topic" > uri="static:(tcp://china:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="china_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="china-queue" > uri="static:(tcp://china:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="china_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="asia-topic" > uri="static:(tcp://asia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="asia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="asia-queue" > uri="static:(tcp://asia:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="asia_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="japan-topic" > uri="static:(tcp://japan:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > dynamicOnly="true"> > <excludedDestinations> > <queue physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <topic physicalName="ActiveMQ.>" /> > <topic physicalName="mcollective.>" /> > <topic physicalName="japan_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > <networkConnector > name="japan-queue" > uri="static:(tcp://japan:61616)" > duplex="true" > decreaseNetworkConsumerPriority="true" > prefetchSize="20000" > dynamicOnly="true" > conduitSubscriptions="false"> > <excludedDestinations> > <topic physicalName=">" /> > </excludedDestinations> > <dynamicallyIncludedDestinations> > <queue physicalName="mcollective.>" /> > <queue physicalName="japan_mcollective.>" /> > </dynamicallyIncludedDestinations> > </networkConnector> > > </networkConnectors> > > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage percentOfJvmHeap="70" /> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb"/> > </storeUsage> > <tempUsage> > <tempUsage limit="1 gb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > <transportConnectors> > <transportConnector name="openwire" uri="tcp://0.0.0.0:61616" > updateClusterClients="true"/> > <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61614"/> > </transportConnectors> > > </broker> > > </beans> > ====================================================================== > > Here is one of the site's activemq.xml file which identical on all sites, > > ====================================================================== > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd > http://activemq.apache.org/camel/schema/spring > http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> > > <!-- Allows us to use system properties as variables in this configuration > file --> > <bean > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="locations"> > <value>file:${activemq.base}/conf/credentials.properties</value> > </property> > </bean> > > <!-- > The <broker> element is used to configure the ActiveMQ broker. > --> > <broker xmlns="http://activemq.apache.org/schema/core"; brokerName="america" > dataDirectory="${activemq.base}/data" networkConnectorStartAsync="true" > schedulePeriodForDestinationPurge="60000"> > > <managementContext> > <managementContext createConnector="false"/> > </managementContext> > > <persistenceAdapter> > <kahaDB directory="${activemq.data}/kahadb"/> > </persistenceAdapter> > > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue=">" producerFlowControl="false" > gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" /> > <policyEntry topic=">" producerFlowControl="false"/> > </policyEntries> > </policyMap> > </destinationPolicy> > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage percentOfJvmHeap="70"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb"/> > </storeUsage> > <tempUsage> > <tempUsage limit="100 mb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > > <transportConnectors> > <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/> > <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/> > </transportConnectors> > > </broker> > > </beans> > ======================================================================== > > here is the wrapper.conf which is identical for all sites included central > one, > > ======================================================================== > # ------------------------------------------------------------------------ > # Licensed to the Apache Software Foundation (ASF) under one or more > # contributor license agreements. See the NOTICE file distributed with > # this work for additional information regarding copyright ownership. > # The ASF licenses this file to You under the Apache License, Version 2.0 > # (the "License"); you may not use this file except in compliance with > # the License. You may obtain a copy of the License at > # > # http://www.apache.org/licenses/LICENSE-2.0 > # > # Unless required by applicable law or agreed to in writing, software > # distributed under the License is distributed on an "AS IS" BASIS, > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > # See the License for the specific language governing permissions and > # limitations under the License. > # ------------------------------------------------------------------------ > > #******************************************************************** > # Wrapper Properties > #******************************************************************** > > #wrapper.debug=TRUE > set.default.ACTIVEMQ_HOME=/usr/share/activemq > set.default.ACTIVEMQ_BASE=/usr/share/activemq > set.default.ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf > set.default.ACTIVEMQ_DATA=%ACTIVEMQ_BASE%/data > wrapper.working.dir=/ > > # Java Application > wrapper.java.command=java > > # Java Main class. This class must implement the WrapperListener interface > # or guarantee that the WrapperManager class is initialized. Helper > # classes are provided to do this for you. See the Integration section > # of the documentation for details. > wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp > > # Java Classpath (include wrapper.jar) Add class path elements as > # needed starting from 1 > wrapper.java.classpath.1=%ACTIVEMQ_HOME%/bin/wrapper.jar > wrapper.java.classpath.2=%ACTIVEMQ_HOME%/bin/activemq.jar > > # Java Library Path (location of Wrapper.DLL or libwrapper.so) > wrapper.java.library.path.1=%ACTIVEMQ_HOME%/bin > > # Java Additional Parameters > # note that n is the parameter number starting from 1. > wrapper.java.additional.1=-Dactivemq.home=%ACTIVEMQ_HOME% > wrapper.java.additional.2=-Dactivemq.base=%ACTIVEMQ_BASE% > wrapper.java.additional.3=-Djavax.net.ssl.keyStorePassword=password > wrapper.java.additional.4=-Djavax.net.ssl.trustStorePassword=password > wrapper.java.additional.5=-Djavax.net.ssl.keyStore=%ACTIVEMQ_CONF%/broker.ks > wrapper.java.additional.6=-Djavax.net.ssl.trustStore=%ACTIVEMQ_CONF%/broker.ts > wrapper.java.additional.7=-Dcom.sun.management.jmxremote > wrapper.java.additional.8=-Dorg.apache.activemq.UseDedicatedTaskRunner=false > wrapper.java.additional.9=-Djava.util.logging.config.file=logging.properties > wrapper.java.additional.10=-Dactivemq.conf=%ACTIVEMQ_CONF% > wrapper.java.additional.11=-Dactivemq.data=%ACTIVEMQ_DATA% > wrapper.java.additional.12=-Djava.security.auth.login.config=%ACTIVEMQ_CONF%/login.config > wrapper.java.additional.13=-Dhawtio.realm=activemq > wrapper.java.additional.14=-Dhawtio.role=admins > wrapper.java.additional.15=-Dhawtio.rolePrincipalClasses=org.apache.activemq.jaas.GroupPrincipal > wrapper.java.additional.16=-Xms16384m > wrapper.java.additional.17=-Xmx16384m > wrapper.java.additional.18=-XX:+UseG1GC > wrapper.java.additional.19=-Dorg.apache.activemq.kahaDB.files.skipMetadataUpdate=true > wrapper.java.additional.20=-XX:MaxMetaspaceSize=16000m > > # Uncomment to enable jmx > #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.port=1616 > #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.authenticate=false > #wrapper.java.additional.n=-Dcom.sun.management.jmxremote.ssl=false > > # Uncomment to enable YourKit profiling > #wrapper.java.additional.n=-Xrunyjpagent > > # Uncomment to enable remote debugging > #wrapper.java.additional.n=-Xdebug -Xnoagent -Djava.compiler=NONE > #wrapper.java.additional.n=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 > > # Initial Java Heap Size (in MB) > #wrapper.java.initmemory=256 > > # Maximum Java Heap Size (in MB) > #wrapper.java.maxmemory=12288 > > # Application parameters. Add parameters as needed starting from 1 > wrapper.app.parameter.1=org.apache.activemq.console.Main > wrapper.app.parameter.2=start > > #******************************************************************** > # Wrapper Logging Properties > #******************************************************************** > # Format of output for the console. (See docs for formats) > wrapper.console.format=PM > > # Log Level for console output. (See docs for log levels) > wrapper.console.loglevel=INFO > > # Log file to use for wrapper output logging. > wrapper.logfile=%ACTIVEMQ_DATA%/wrapper.log > > # Format of output for the log file. (See docs for formats) > wrapper.logfile.format=LPTM > > # Log Level for log file output. (See docs for log levels) > wrapper.logfile.loglevel=INFO > > # Maximum size that the log file will be allowed to grow to before > # the log is rolled. Size is specified in bytes. The default value > # of 0, disables log rolling. May abbreviate with the 'k' (kb) or > # 'm' (mb) suffix. For example: 10m = 10 megabytes. > wrapper.logfile.maxsize=0 > > # Maximum number of rolled log files which will be allowed before old > # files are deleted. The default value of 0 implies no limit. > wrapper.logfile.maxfiles=0 > > # Log Level for sys/event log output. (See docs for log levels) > wrapper.syslog.loglevel=NONE > > #******************************************************************** > # Wrapper Windows Properties > #******************************************************************** > # Title to use when running as a console > wrapper.console.title=ActiveMQ > > #******************************************************************** > # Wrapper Windows NT/2000/XP Service Properties > #******************************************************************** > # WARNING - Do not modify any of these properties when an application > # using this configuration file has been installed as a service. > # Please uninstall the service before modifying this section. The > # service can then be reinstalled. > > # Name of the service > wrapper.ntservice.name=ActiveMQ > > # Display name of the service > wrapper.ntservice.displayname=ActiveMQ > > # Description of the service > wrapper.ntservice.description=ActiveMQ Broker > > # Service dependencies. Add dependencies as needed starting from 1 > wrapper.ntservice.dependency.1= > > # Mode in which the service is installed. AUTO_START or DEMAND_START > wrapper.ntservice.starttype=AUTO_START > > # Allow the service to interact with the desktop. > wrapper.ntservice.interactive=false > ============================================================================= > > > Regards > Ravi > > >> On Saturday, March 4, 2017 at 11:31:49 AM UTC+5:30, R.I.Pienaar wrote: >> You'll need to give some details on how activemq is configured >> >> mco ping does not use the --nodes argument. Try mco rpc rpcutil ping. That >> will use it. >> >> --- >> R.I.Pienaar >> >>> On 4 Mar 2017, at 05:29, [email protected] wrote: >>> >>> Hi, >>> >>> I want to use mcollective on geographically and attempted to configure >>> ActiveMQ network of brokers for that. After the configuration everything is >>> working fine as expected but only problem is, when I run any mco query for >>> multiple servers then its getting disconnected for sometime when many of >>> the not responded. >>> >>> For example, I have connected 1000 clients for each location of ActiveMQ >>> broker, and am deploying mcollective on many servers and I have the entire >>> list of servers(458) which am deploying it. So when I run mco ping or any >>> mco queries with --nodes=<server_list_file> geographically am getting >>> response from 243 servers which means mcollective configured successfully >>> on these servers and no response from other servers which means mcollective >>> not configured yet on those servers. But the problem is here many servers >>> are not responded for mcollective and due to this when I run the mco query >>> on connected servers am not getting response. But If I leave for few >>> minutes and then run the same query against working servers then am getting >>> response. Am suspecting that when I run mco query and many servers are not >>> responded then middleware takes time to clear pending queues or something >>> happening. Any idea to tune this? >>> >>> >>> Regards >>> Ravi >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google Groups >>> "mcollective-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "mcollective-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
