Hi, just to clarify. In my last post, when I mentioned 'all the pending messages are in memory', I actually mean uncommitted messages within a transaction. Again, can someone confirm if this statement (all uncommitted messages are kept in memory) is true?
aliu wrote: > > The reason I've been trying to figure out these memory limit > configurations is because we have the following use case: > > I want to be able to send a large amount of data (>2g) to the queue > without any consumer consuming it for some time, but it seems like active > mq actually keeps all the pending messages in memory, I just want to > confirm this observation since this would put a lot of constraint on our > memory requirement. > > I are using persistent messaging with transactions, with memoryUsage set > to 1g, when sending over 50mb of data, I get an OutOfMemory error in the > middle (around 40mb). But when I tried to do a commit with every 3mb of > data, there was no errors. > > I just want to confirm whether active mq actually keeps all the pending > messages in memory. > > Thanks, > Audrey > > > > ammulder wrote: >> >> Are you validating the XML just to make sure it's not a syntax error? >> >> I get stuff onto disk with a configuration like this (and a producer >> sending 1 MB messages): >> >> <destinationPolicy> >> <policyMap> >> <policyEntries> >> <policyEntry queue=">" memoryLimit="50mb"/> >> <policyEntry topic=">" memoryLimit="50mb"> >> ... >> <systemUsage> >> <systemUsage> >> <memoryUsage> >> <memoryUsage limit="20 mb"/> >> </memoryUsage> >> ... >> >> >> Thanks, >> Aaron >> >> On Tue, Jul 15, 2008 at 10:50 AM, aliu <[EMAIL PROTECTED]> wrote: >>> >>> Hi, >>> Thanks for the quick reply. If I don't want to keep everything in >>> memory, >>> and I want to be able to set how much should be stored in memory and if >>> there are more stuff coming, it'd be spilled out to disk, what >>> configuration >>> settings can I use? >>> >>> I actually tried setting the destination limits high and the system >>> limit >>> low, with the following config: >>> <policyEntry queue=">" memoryLimit="1 gb"/> >>> ... >>> <memoryUsage> >>> <memoryUsage limit="30 mb"/> >>> </memoryUsage> >>> >>> When I'm trying to send 50mb worth of data, my producer blocks after >>> sending >>> about 30mb of data. And I also tried with destination limits low and >>> system >>> limit high, my producer also blocks after sending the lower limit of the >>> two >>> settings. >>> >>> Just wondering if anyone can give me some advice on what I can set on >>> how >>> much to be allowed in memory? >>> >>> Thanks in advance. >>> Audrey >>> >>> ammulder wrote: >>>> >>>> I am by no means an authority on this, but I'm not sure the store >>>> usage or temp usage settings are actually used for anything. >>>> >>>> I believe you want your memory usage setting to be high enough to hold >>>> all the traffic you expect to be in memory at once. I think this is >>>> e.g. non-persistent messages that have not yet been delivered (regular >>>> in-flight messages, queues that are backed up, topics with durable >>>> subscribers who are away, etc.). There are separate settings that can >>>> be applied to the memory limit for each destination (or wildcard >>>> groups of queues/topics, etc.). >>>> >>>> If you set the destination limits high and the system limit low, you >>>> can cause ActiveMQ to start using disk space (as "swap" I guess), but >>>> I hear that's prone to deadlocks and etc. -- plus it doesn't seem >>>> bound by the store/temp usage settings. >>>> >>>> If you are up against memory limits (due to slow/disconnected >>>> consumers or whatever), there are various additional considerations >>>> and configuration options you may want to use. >>>> >>>> Thanks, >>>> Aaron >>>> >>>> On Mon, Jul 14, 2008 at 6:47 PM, aliu <[EMAIL PROTECTED]> wrote: >>>>> >>>>> Hi, I am a new user of the ActiveMQ and I was trying to figure out how >>>>> to >>>>> tweak different memory settings. >>>>> I read on the forum that storeUsage controls the maximum size of the >>>>> AMQMessageStore, and memoryUsage is the maximum amount of memory the >>>>> broker >>>>> will use. >>>>> >>>>> However, when I ran a test of sending 50,000 messages with 1k for each >>>>> message, and I've used the following config: >>>>> <systemUsage> >>>>> <systemUsage> >>>>> <memoryUsage> >>>>> <memoryUsage limit="30 mb"/> >>>>> </memoryUsage> >>>>> <storeUsage> >>>>> <storeUsage limit="1 gb" name="foo"/> >>>>> </storeUsage> >>>>> <tempUsage> >>>>> <tempUsage limit="1 mb"/> >>>>> </tempUsage> >>>>> </systemUsage> >>>>> </systemUsage> >>>>> >>>>> My producer would block after sending about 30mb worth of data, even >>>>> though >>>>> I have specified the storeUsage to be big enough to contain all my >>>>> messages. >>>>> I thought what it would be is to have 30mb worth of data in memory, >>>>> then >>>>> it'll spill over once it exceeds the limit. >>>>> >>>>> Can someone please tell me in detail how these settings work together? >>>>> I >>>>> am >>>>> at a loss here. >>>>> >>>>> I am using 5.1, persistent messaging with transactions. >>>>> >>>>> Thanks in advance. >>>>> Audrey >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html >>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>> >>>>> >>>>> >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/systemUsage-configuration-tp18454843p18467144.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >>> >>> >> >> > > -- View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18490842.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
