On 09/16/2011 03:11 PM, Jeff Armstrong wrote:
I tried increasing the message size to 1K and found that the overhead per
message was indeed the same. So this leads me to the next questions: why is the
overhead so large per message, and is there a way to reduce this? From what I
can tell from the client side of things, each message has a bunch of bool/int
fields and a few small strings (routing key, exchange name, etc.), but nothing
that would put it up over 1K.
I don't think anyones done a detailed analysis of qpid's per-message memory
overhead. It would be interesting to run a tool like valgrind on the broker
and see where the memory is going. There is probably room for improvement.
________________________________________
From: Kim van der Riet [[email protected]]
Sent: Friday, September 09, 2011 2:34 PM
To: [email protected]
Subject: RE: qpidd using approx 10x memory
On Fri, 2011-09-09 at 12:19 -0400, Jeff Armstrong wrote:
So I just tried my test again after compiling the broker with the following
code added and still saw the same issue of 10x memory consumption.
mallopt(M_ARENA_TEST, 0);
mallopt(M_ARENA_MAX, 1);
________________________________________
From: Jeff Armstrong [[email protected]]
Sent: Friday, September 09, 2011 11:40 AM
To: [email protected]
Subject: RE: qpidd using approx 10x memory
The sample code I provided only publishes messages - there is no consuming
going on, so this doesn't follow the pattern you are suggesting. If there is no
freeing happening because there is no consumer, it seems like there must be
some other cause of this.
Also, in my application, there are two connections publishing and 2+
connections consuming and this shows the same problem.
I re-read your original mail, and I think I misunderstood your issue,
sorry.
My guess is that the per-message overhead for each message is much
larger than the message itself in your case - 100 bytes is a rather
small message. I have always assumed a 1-2KiB memory overhead per
message for headers and other bookkeeping purposes.
One way to check this is to increase the size of the message only to,
say, 1KiB or 10KiB, and check if the overhead remains constant.
Perhaps others can comment?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
