Hello, We have the requirement of creating a very large number of queues (tens of thousands) in a single broker. My searches turned up several others trying to do something similar, but with no results. We were recently able to accomplish this, so I decided to write a little howto. All of this information can be found in the mailing list, wiki, or XML reference, but it wasn't easy to pull it all together. Hopefully this will help someone. Also, please let me know if there are any inaccuracies, or if anything can be added.
== Thread Count == === "QueueThread" === One of the issues that you'll come across is receiving the out of memory error "unable to create new native thread" when you create thousands of queues. A "QueueThread" thread is created for every new queue. The optimizedDispatch property for the queue policyEntry seemed to address our needs, the description states "don't use a separate thread for dispatching from a Queue" (http://activemq.apache.org/per-destination-policies.html). However, turning this on just seemed to create "TempQueue" threads instead, still creating a thread for each queue. Digging a little deeper, we find that using OptimizedDispatch creates the thread using a TaskRunner, which should pool the threads. After further research, we found that the property org.apache.activemq.UseDedicatedTaskRunner should be set to false so that the task runner actually pools the threads (http://activemq.apache.org/javalangoutofmemory.html --- this document also provides other useful info, like how to lower the stack size for each thread). Example Policy Entry in INSTALL_DIR/conf/activemq.conf: <policyEntries> <policyEntry queue=">" memoryLimit="15mb" optimizedDispatch='true'/> </policyEntries> You can either change the 'UseDedicatedTaskRunner' property in your startup script, INSTALL_DIR/bin/activemq: ACTIVEMQ_OPTS="-Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=false" Or you can set ACTIVEMQ_OPTS in /etc/activemq.conf . Note: It would be very useful it the documentation listed above referenced each other, as they are most useful when used together. === "Checkpoint" thread === If you're using persistent storage, a "Checkpoint" thread is also created for every queue that is created. These are used to write persistent messages to the store. Fortunately this is already created with a task runner, so setting the UseDedicatedTaskRunner property to false will prevent these thread from overhwelming your system. === Connection related threads === If you're also dealing with high thread count due to a very high number of connections, see http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ. Also, consider reducing the stack size for each thread (the JVM option -Xss). == "too many open files" == ActiveMQ uses the amqPersistenceAdapter by default for persistent messages. Unfortunately, this persistence adapter (as well as the kahaPersistenceAdapter) opens a file descriptor for each queue. When creating large numbers of queues, you'll quickly run into the limit for your OS. There are two possible solutions to this: 1.) raise the per-process open file limit for your OS 2.) Use a jdbc persistent adapter. The first can simply be googled for your OS. The second uses a connection pool to your DB of choice, and no longer requires an open file for each queue created. An unrelated issue is running out of file descriptors because there are too many connections. (In linux, and other OSes, a file descriptor is used for open sockets). I don't know of any other way to handle that, other than increasing the limit for your process. == Conclusion == We were able to create more than 20k queues using ~30 threads in a test environment with a 512MB heap. Environment: ActiveMQ 5.2.0, SUN Java 1.6.0, Ubuntu 9.04 . Our testing hasn't been exhaustive, but everything is running smoothly so far. JLuna
