> Could you please, clarify me does paging can be tuned for bath-retrievig messages form disk for the faster message processing.
The method you referenced as getting "messages from disk one by one" (i.e. org.apache.activemq.artemis.core.paging.impl.Page.read(StorageManager)) isn't doing quite what you describe. It isn't reading a single _message_ from disk; it's reading an entire _page_ from disk. Each page consists of multiple messages (configurable via page-size-bytes). Take a look at the paging documentation [1] for more details. > And, in general, is paging a good solution for my case, may be there are some better solutions for solving my problem? You might be better served by using flow-control to throttle the producers and/or using the BLOCK address-full-policy. At the end of the day if you produce messages faster than you consume them you're going to eventually run out of resources on the broker. You'll hit your memory limit first, and then (if paging is configured) eventually you'll hit your disk limit. The only real way to recover from such a situation is to slow down production or increase consumption such that the messages consumed "catches up" to the messages that have been produced. You can increase performance by giving the broker more memory (i.e. larger address-full-size), but anytime you start paging you'll be working with a _much_ slower medium than RAM (whether that's spinning HDDs or SSDs). Justin [1] http://activemq.apache.org/artemis/docs/latest/paging.html On Sun, Oct 1, 2017 at 5:07 AM, Sergey Zagalskiy <[email protected]> wrote: > Hi, Artemis team! > I have some problem with Artemis paging in my highload application. > In a nutshell, in my application producers produces messages very often - > about 500 messages per second. > At the same time, consumer is sometimes pretty slow. So consumer processess > messages slower than producer produces them. In this case why messages > accumulate in queue, clog it up and application fails with > OutOfMemoryError. > I'm trying to use paging for "mitigating" this problem. As a result, > application doesn't fail with OutOfMemoryError, messages are stored on disk > - not in memory. > But... I have an issue here. If application paged a lot of messages (100k) > once - consumer can not process messages from paging and "new" messages > from > queue. Even if consumer in very fast - even if he doesn't do anything > during > processing. > So, if application jumped to paging once - it will never recover its > initial > message processing speed. > > This is how I test it on my local environment: > I have following jms architecture > > Producer => QueueOUT => Bridge => QueueIN => Consumer. > > Consumer DOES NOT DO ANYTHING. This is just a listener for queue with blank > method. > > I stop the bridge. > Wait until QueueOUT will have 100k messages > Start a bridge - messages are transfered to QueueIN and Consumers starts to > process them. > At the same time producer produces new messages, which are transfered to > QueueIN. But consumer can not proccess "old" messages form paging and "new" > messages, which just delivered from producer. > > As I understood from investigation of the paging code - method > org.apache.activemq.artemis.core.paging.impl.Page.read(StorageManager) > Artemis gets messages from disk one by one this algorithm is hardcoded. And > there are no settings to be configured for Artemis batch retrieving > messages > from disk. > > Could you please, clarify me does paging can be tuned for bath-retrievig > messages form disk for the faster message processing. > And, in general, is paging a good solution for my case, may be there are > some better solutions for solving my problem? > > Thanks in advance for your help > Sergey > > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev- > f2368404.html >
