Hi again: Thank you for your reply.
By "stuck" I meant literally stuck. No messages were being processed. For instance, below is the queue stats of Queue#1 Pending No. of Consumers Enqueued Dequeued 8016495 1 18094028 10077534 The de-queuing process was stuck. So, was the de-queuing process of Queue#2 Karaf logs did not show any signs of issues. Thus, after looking at broker logs, I found the following log message (as I shared earlier): > Usage(default:store:queue://tst.tst.users:store) percentUsage=99%, > usage=42792917913, limit=40692963880, > percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%, > usage=42792917913, limit=40692963880, percentUsageMinDelta=1%: Persistent > store is Full, 100% of 40692963880. Stopping producer > (ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding > queue://tst.tst.users. See > http://activemq.apache.org/producer-flow-control.html for more info > (blocking for: 228222s) | org.apache.activemq.broker.region.Queue | > ActiveMQ > Transport: tcp:///192.168.7.98:51461@61616 Right now, around 200-300 messages get de-queued from Queue#1, while just about 100-150 messages get de-queued from Queue#2. That's why I assumed that the bottleneck could have been the database endpoint. The number of queues in this broker will grow significantly in upcoming days, with the increasing number of my karaf instances. But, the issue that I experienced about producer-flow-control is specific to the limit of a queue but not with the number of queues, right? So far, in the broker optimization process, I have only enabled optimizedDispatch. I am yet to measure the performance benefits of usage of NIO. Regards, Cooshal. On Tue, Jul 2, 2019 at 3:13 AM Tim Bain <tb...@alumni.duke.edu> wrote: > I had assumed that messages were being processed, just not as fast as you > wanted (and the producer got limited to the rate at which the consumer > processed them), but you said the word "stuck" several times. Which is it? > Are you not processing messages, or are you not processing messages > quickly? I want to make sure we understand the behavior you're actually > observing, since that will influence any advice we may be able to give you. > > You asked how to change the producer flow control limit for the persistent > store. The configuration snippet you referenced is the correct way to set > that. > > Your description of your processing flow and the fact that your backup > occurs on queue #1 means that the write to the database is not your > performance limiter, because if it was you'd see a backup on queue #2. > Rather, your consumer that translates messages from queue #1 and writes > them to queue #2 seems like it's not keeping up, because if it was then > messages written to queue #1 wouldn't back up there (instead they'd back up > on queue #2). > > Tim > > On Mon, Jul 1, 2019, 8:25 AM <kushal.gau...@gmail.com> wrote: > >> Hi: >> >> thank you for your reply. >> >> My messages to the forum are being bounced back. Thus, I am sending this >> directly to you. >> >> Option 1 (do nothing and wait) did nothing, as all my messages were stuck >> in-flight, and I realized that this was stuck for more than 7 hours. >> Restarting camel contexts or instances did not help either. >> >> Yes, there is a big gap between the rates of consumption and production. >> My assumption is that the bottleneck due to the database endpoint. I am not >> able to profile this behaviour. >> >> My current process is: >> >> DB Consumer (from database) -> queue 1 >> Queue 1 consumer (consume from queue 1) -> transform the data -> queue 2 >> Queue 2 consumer (consume from queue 2) -> write to DB >> >> and this has resulted in millions of records to be stuck in queue 1 as >> pending. >> >> I have restarted the broker, and it started consuming these messages now. >> >> How do I increase the limit for persistent store? >> >> <storeUsage> >> <storeUsage limit="100 gb"/> >> </storeUsage> >> >> Is this the right config to change? >> >> Regards, >> Cooshal. >> >> <quote author='Tim Bain'> >> It appears that your consumer isn't keeping up with your producer(s), >> filling the persistent store, which slows your producer(s) to the rate at >> which the consumer is consuming. >> >> Your options are: >> * Do nothing, and let it complete at the current rate. >> * Speed up consumption, either by adding consumers or making them faster >> or >> both. >> * Increase the limit for the persistent store to allow the producer to >> write all remaining messages to the broker, so that the producer process >> can exit even though the consumer is still processing messages. Note that >> this requires a broker restart, so make sure your producer logic will >> gracefully handle a period of broker unavailability. Persistent messages >> will remain on the broker when a restart occurs, and since it's the >> persistent store that is full, all of those messages will be kept, but if >> you have any unconsumed non-persistent messages, they'll be lost. >> >> Tim >> >> On Mon, Jul 1, 2019, 5:51 AM Kushal Gautam <kushal.gau...@gmail.com> >> wrote: >> >> > Hi: >> > >> > I have similar issues ( >> > >> > >> http://activemq.2283324.n4.nabble.com/Persistent-store-is-Full-100-of-107374182400-Stopping-producer-td4740024.html >> > ). >> > My activemq broker was mediating the transfer of millions of records, >> > which >> > I was importing via Apache Camel (running on a >> > Karaf instance). >> > >> > I am currently using ActiveMQ 5.15.4, and below is how my systemUsage >> > config >> > looks like: >> > >> > <systemUsage> >> > <systemUsage> >> > <memoryUsage> >> > <memoryUsage percentOfJvmHeap="70" /> >> > </memoryUsage> >> > <storeUsage> >> > <storeUsage limit="100 gb"/> >> > </storeUsage> >> > <tempUsage> >> > <tempUsage limit="50 gb"/> >> > </tempUsage> >> > </systemUsage> >> > </systemUsage> >> > >> > This is what I got in the logs: >> > >> > Usage(default:store:queue://tst.tst.users:store) percentUsage=99%, >> > usage=42792917913, limit=40692963880, >> > percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%, >> > usage=42792917913, limit=40692963880, percentUsageMinDelta=1%: >> Persistent >> > store is Full, 100% of 40692963880. Stopping producer >> > (ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding >> > queue://tst.tst.users. See >> > http://activemq.apache.org/producer-flow-control.html for more info >> > (blocking for: 228222s) | org.apache.activemq.broker.region.Queue | >> > ActiveMQ >> > Transport: tcp:///192.168.7.98:51461@61616 >> > >> > I have restarted the karaf instances, but this does not seem to take an >> > effect, and after looking at the broker logs, it quite made sense. >> > >> > What can I actually do to avert this problem? >> > >> > Currently, the broker stats for my queue shows: >> > Pending No. of Consumers Enqueued Dequeued >> > 8016495 1 18094028 10077534 >> > >> > I am not sure, if restarting the broker will result in loss of these >> > messages. >> > >> > Also, would be great if you could suggest about the config options that >> I >> > can make to optimize it better. >> > >> > Regards, >> > Cooshal. >> > >> >> </quote> >> Quoted from: >> >> http://activemq.2283324.n4.nabble.com/ActiveMQ-persistence-store-is-full-Stopping-producer-tp4751371p4751374.html >> >> >> _____________________________________ >> Sent from http://activemq.2283324.n4.nabble.com >> >> > On Mon, Jul 1, 2019, 8:25 AM <kushal.gau...@gmail.com> wrote: > >> Hi: >> >> thank you for your reply. >> >> My messages to the forum are being bounced back. Thus, I am sending this >> directly to you. >> >> Option 1 (do nothing and wait) did nothing, as all my messages were stuck >> in-flight, and I realized that this was stuck for more than 7 hours. >> Restarting camel contexts or instances did not help either. >> >> Yes, there is a big gap between the rates of consumption and production. >> My assumption is that the bottleneck due to the database endpoint. I am not >> able to profile this behaviour. >> >> My current process is: >> >> DB Consumer (from database) -> queue 1 >> Queue 1 consumer (consume from queue 1) -> transform the data -> queue 2 >> Queue 2 consumer (consume from queue 2) -> write to DB >> >> and this has resulted in millions of records to be stuck in queue 1 as >> pending. >> >> I have restarted the broker, and it started consuming these messages now. >> >> How do I increase the limit for persistent store? >> >> <storeUsage> >> <storeUsage limit="100 gb"/> >> </storeUsage> >> >> Is this the right config to change? >> >> Regards, >> Cooshal. >> >> <quote author='Tim Bain'> >> It appears that your consumer isn't keeping up with your producer(s), >> filling the persistent store, which slows your producer(s) to the rate at >> which the consumer is consuming. >> >> Your options are: >> * Do nothing, and let it complete at the current rate. >> * Speed up consumption, either by adding consumers or making them faster >> or >> both. >> * Increase the limit for the persistent store to allow the producer to >> write all remaining messages to the broker, so that the producer process >> can exit even though the consumer is still processing messages. Note that >> this requires a broker restart, so make sure your producer logic will >> gracefully handle a period of broker unavailability. Persistent messages >> will remain on the broker when a restart occurs, and since it's the >> persistent store that is full, all of those messages will be kept, but if >> you have any unconsumed non-persistent messages, they'll be lost. >> >> Tim >> >> On Mon, Jul 1, 2019, 5:51 AM Kushal Gautam <kushal.gau...@gmail.com> >> wrote: >> >> > Hi: >> > >> > I have similar issues ( >> > >> > >> http://activemq.2283324.n4.nabble.com/Persistent-store-is-Full-100-of-107374182400-Stopping-producer-td4740024.html >> > ). >> > My activemq broker was mediating the transfer of millions of records, >> > which >> > I was importing via Apache Camel (running on a >> > Karaf instance). >> > >> > I am currently using ActiveMQ 5.15.4, and below is how my systemUsage >> > config >> > looks like: >> > >> > <systemUsage> >> > <systemUsage> >> > <memoryUsage> >> > <memoryUsage percentOfJvmHeap="70" /> >> > </memoryUsage> >> > <storeUsage> >> > <storeUsage limit="100 gb"/> >> > </storeUsage> >> > <tempUsage> >> > <tempUsage limit="50 gb"/> >> > </tempUsage> >> > </systemUsage> >> > </systemUsage> >> > >> > This is what I got in the logs: >> > >> > Usage(default:store:queue://tst.tst.users:store) percentUsage=99%, >> > usage=42792917913, limit=40692963880, >> > percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=105%, >> > usage=42792917913, limit=40692963880, percentUsageMinDelta=1%: >> Persistent >> > store is Full, 100% of 40692963880. Stopping producer >> > (ID:xxx-yy-0066-51437-1561640932972-1:24:3:1) to prevent flooding >> > queue://tst.tst.users. See >> > http://activemq.apache.org/producer-flow-control.html for more info >> > (blocking for: 228222s) | org.apache.activemq.broker.region.Queue | >> > ActiveMQ >> > Transport: tcp:///192.168.7.98:51461@61616 >> > >> > I have restarted the karaf instances, but this does not seem to take an >> > effect, and after looking at the broker logs, it quite made sense. >> > >> > What can I actually do to avert this problem? >> > >> > Currently, the broker stats for my queue shows: >> > Pending No. of Consumers Enqueued Dequeued >> > 8016495 1 18094028 10077534 >> > >> > I am not sure, if restarting the broker will result in loss of these >> > messages. >> > >> > Also, would be great if you could suggest about the config options that >> I >> > can make to optimize it better. >> > >> > Regards, >> > Cooshal. >> > >> >> </quote> >> Quoted from: >> >> http://activemq.2283324.n4.nabble.com/ActiveMQ-persistence-store-is-full-Stopping-producer-tp4751371p4751374.html >> >> >> _____________________________________ >> Sent from http://activemq.2283324.n4.nabble.com >> >>
