[
https://issues.apache.org/jira/browse/AMQ-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654661#comment-16654661
]
Alan Protasio commented on AMQ-6590:
------------------------------------
Hum....
I dont really think that this is solving the all problem.... Of course in the
case of a crash (unclean shutdown) the fail over will be faster....
But after that, let's imagine that we wanna do a normal restart. Well the
broker will have a long shutdown and while doing that it will still hold the
lock making the failover slow the same way.
And maybe now is worst... Imagine that the shutdown is taking 2 minutes and the
ACTIVEMQ_KILL_MAXSECONDS is set to 30 secs...
In this case all the shutdown will be unclean (so lots of dangling free pages -
larger index file on each restart) until we delete the index file and perform a
full recovery.
Why don't we save the free page in every checkpoint + keep track of it's hash
or fingerPrint (SequenceSetHash) in the metadata (the page size is already
saved there).
[https://github.com/apache/activemq/blob/5d3a3fcca7fbc04e60a146e42fb1f65b94e4ea7b/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/page/PageFile.java#L468]
Now, in the load method, we can check if the fingerprint of the metadata is
equal to the saved free list... if it's equal we can load this list, if it's
not, we can recover it (read the whole index file as before).
So in each checkpoint we should:
Calculate freeList hash (or this can be pre calculated changing it every time
that this list is modified)
Save the free list
Save this information in the metadata
Now, on the load we can check if the hash of the loaded free list is the equal
to the metadata... if it is we are good.
If you think that this makes sense i can create a Jira to try to implement
this..
This is a rough draft of what I meant and seems to be working.
[https://github.com/alanprot/activemq/commit/18036ef7214ef0eaa25c8650f40644dd8b4632a5]
Is important to note that in the Pagefile#load method we need to rebuild the
index in the same way it was in the checkpoint (the recovery will be performed
after to replay the time btw the checkpoint and the crash)
So...
Imagine the timeline:
T0 -> P1, P2 and P3 are free.
T1 -> Checkpoint
T2 -> P1 got occupied.
T3 -> Crash
In the current scenario after the Pagefile#load the P1 will be free and then
the replay will mark P1 as occupied...
My change only make sure that db.data and db.free are in sync and showing the
reality in T1 (checkpoint), If they are in sync i can use the db.free.
> KahaDB index loses track of free pages on unclean shutdown
> ----------------------------------------------------------
>
> Key: AMQ-6590
> URL: https://issues.apache.org/jira/browse/AMQ-6590
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.14.3
> Reporter: Christopher L. Shannon
> Assignee: Christopher L. Shannon
> Priority: Major
> Fix For: 5.15.0, 5.14.4, 5.16.0, 5.15.7
>
>
> I have discovered an issue with the KahaDB index recovery after an unclean
> shutdown (OOM error, kill -9, etc) that leads to excessive disk space usage.
> Normally on clean shutdown the index stores the known set of free pages to
> db.free and reads that in on start up to know which pages can be re-used. On
> an unclean shutdown this is not written to disk so on start up the index is
> supposed to scan the page file to figure out all of the free pages.
> Unfortunately it turns out that this scan of the page file is being done
> before the total page count value has been set so when the iterator is
> created it always thinks there are 0 pages to scan.
> The end result is that every time an unclean shutdown occurs all known free
> pages are lost and no longer tracked. This of course means new free pages
> have to be allocated and all of the existing space is now lost which will
> lead to excessive index file growth over time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
