Maintaining both strategies sounds like a good idea (and a fun exercise so I might give it a go over the weekend :p).
Jira item created: https://issues.apache.org/jira/browse/PROTON-597 -- Marcel On Fri, Jun 6, 2014 at 4:00 PM, Rafael Schloming <[email protected]> wrote: > That sounds like a worthy improvement to me. We could possibly put in a > threshold to let people choose their own tradeoff, e.g. below X buffer > size we use one strategy, and above it we use another. Either way, I'd go > ahead and file a JIRA with a pointer to your commit. > > --Rafael > > > On Thu, Jun 5, 2014 at 4:48 PM, Marcel Meulemans < > [email protected]> wrote: > > > I have been experiencing high (resident) memory usage when using ActiveMQ > > with large amounts of AMQP connections. > > > > After some investigation I have traced this memory usage back to > proton-j. > > The cause is that by default ActiveMQ defines a maxFrameSize of 1MB for > the > > AmqpWireFormat and in proton-j there are two classes > (TrasportOutputAdaptor > > and FrameParser) that hold a ByteBuffer of maxFrameSize throughout their > > lifetime. In the case of the 1MB frame size that means that each > transport > > has a fairly high memory footprint (2MB) considering that this memory is > > hardly ever used, i.e. only when there is a frame to transfer in/out and > in > > most cases the frame will not be maxFrameSize. Because there is a > Transport > > for every connection the memory usage can become quite high when dealing > > with many connections with large maxFrameSize, e.g. 2GB for only 1000 > > connections (I was aiming for 10000 connections so you see my problem). > > > > Generally the buffer allocation may not be a problem but in my case they > > are so I decided to modify proton so that it does not require these > buffers > > to exist at all times. The change I made are a classical tradeoff between > > memory usage and cpu cycles. The memory usage has been reduced and the > cpu > > usage increased (more frequent allocations). The buffers are only > allocated > > when needed and are released as soon as they hold no more data. The > changes > > (which pass all tests and have the desired results) can be found here: > > https://github.com/marcelmeulemans/qpid-proton > > > > Because this may not be considered a bug or even an improvement I am > > sharing it here so you guys can decide is this is in anyway useful. > > > > -- > > Marcel > > >
