That sounds like a worthy improvement to me. We could possibly put in a threshold to let people choose their own tradeoff, e.g. below X buffer size we use one strategy, and above it we use another. Either way, I'd go ahead and file a JIRA with a pointer to your commit.
--Rafael On Thu, Jun 5, 2014 at 4:48 PM, Marcel Meulemans < [email protected]> wrote: > I have been experiencing high (resident) memory usage when using ActiveMQ > with large amounts of AMQP connections. > > After some investigation I have traced this memory usage back to proton-j. > The cause is that by default ActiveMQ defines a maxFrameSize of 1MB for the > AmqpWireFormat and in proton-j there are two classes (TrasportOutputAdaptor > and FrameParser) that hold a ByteBuffer of maxFrameSize throughout their > lifetime. In the case of the 1MB frame size that means that each transport > has a fairly high memory footprint (2MB) considering that this memory is > hardly ever used, i.e. only when there is a frame to transfer in/out and in > most cases the frame will not be maxFrameSize. Because there is a Transport > for every connection the memory usage can become quite high when dealing > with many connections with large maxFrameSize, e.g. 2GB for only 1000 > connections (I was aiming for 10000 connections so you see my problem). > > Generally the buffer allocation may not be a problem but in my case they > are so I decided to modify proton so that it does not require these buffers > to exist at all times. The change I made are a classical tradeoff between > memory usage and cpu cycles. The memory usage has been reduced and the cpu > usage increased (more frequent allocations). The buffers are only allocated > when needed and are released as soon as they hold no more data. The changes > (which pass all tests and have the desired results) can be found here: > https://github.com/marcelmeulemans/qpid-proton > > Because this may not be considered a bug or even an improvement I am > sharing it here so you guys can decide is this is in anyway useful. > > -- > Marcel >
