On 3/19/20 2:33 PM, Clebert Suconic wrote:
Can we stop doing that check?
If we are out of memory, we could certainly write non durable messages
to disk, but that doesn't mean they became durable.
That has always been my thought in this, it is not suddenly durable
simply because it was paged.
On Thu, Mar 19, 2020 at 12:21 PM Timothy Bish <[email protected]> wrote:
On 3/19/20 9:43 AM, nigro_franz wrote:
HI folks,
I'm thinking how to improve (the performance and stability) of AMQP paging
and I've fallen into this behaviour for paging AMQ standard messages:
https://github.com/apache/activemq-artemis/blob/d5104656f9060d347b326c59e136dac0b3c404e0/artemis-protocols/artemis-amqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java#L840
This reencode() is a very intensive operation and is being triggered on
depaging (exactly here
https://github.com/apache/activemq-artemis/blob/51c2022f388a5c70adbc5de6b799f9072960c2f1/artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/cursor/PagedReferenceImpl.java#L154).
It force non durable AMQP messages to see their AMQP header to be created,
if not present, just to set that property: given that the header part of
AMQP messages is encoded at the beginning of the message, it forces a
reencode of the whole message.
I suppose this could be improved in different ways, but I was thinking to
always preset an header with durable == false in ((when appropriate, with
non-durable messages): it would make every non durable messages a lil
bigger, but will save lot of memory and CPU time on depaging, because the
reencode could be optimized and be partial (reencoding only the few bits
necessary on the header, now present).
If the broker is going to update the message as it comes out of the
journal to indicate it has become durable then why not beat that to the
punch by just putting it in there that way when there no Header or the
existing Header is marked as non-durable?
* In the case of not having a header you'd then only need to first write
one and then the remaining already encoded message portion.
* In the case that there was a Header I think the AMQPMessage tracks
those location so you could update and re-encode only the Header and
then do as above and just write out the remainder of the previously
encoded message.
I wouldn't just add a Header to every message though as the broker
really should just relay the message as is if it didn't need to store it
in the first place. I've always been a bit dubious on the whole marking
it durable just because it was paged thing but that might just be me.
I'm opened to discuss this and find together a smart way to improve
depaging: given that it happens when the JVM memory is supposed to be
precious I believe that saving time/memory there is crucial not just for
performance but for the broker stability.
Cheers,
Franz
I've already opened&solved
https://issues.apache.org/jira/browse/ARTEMIS-2661 to improve AMQP journal
loading by saving scanning AMQP message data if not strictly necessary and
it already deliver exceptional result: I would like to do the same on
paging, given that this really seems a low hanging fruit with an high
potential of improvement.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
--
Tim Bish
--
Tim Bish
