Maybe someone else has an explanation.. but wonder if the index and transaction log became out of sync somehow and a restart kicked it in the pants sufficiently? There has been a lot of hardening around KahadB since 5.5.1.. Can you reproduce it in any reliable way?
On Tue, Jul 23, 2013 at 11:18 AM, Greenbean <[email protected]> wrote: > We have encountered a problem where kaha files are build up on occasion. > Restarting AMQ seems to allow cleanup to occur. There is a log entry > before > the restart which explains contained ACKs are preventing cleanup. Just > curious how the restart works around this contained ack issue. > > Example recent persisted message files on a server: > 33M -rw-r--r-- 1 sysuser sysuser 33M Jul 9 17:12 db-62483.log > 44M -rw-r--r-- 1 sysuser sysuser 44M Jul 9 17:16 db-62484.log > 56M -rw-r--r-- 1 sysuser sysuser 56M Jul 9 17:20 db-62485.log > 36M -rw-r--r-- 1 sysuser sysuser 36M Jul 9 17:34 db-62486.log > 0 -rw-r--r-- 1 sysuser sysuser 0 Jul 12 02:43 lock > 33M -rw-r--r-- 1 sysuser sysuser 33M Jul 12 02:47 db-62869.log > 52M -rw-r--r-- 1 sysuser sysuser 52M Jul 12 02:47 db-62870.log > 33M -rw-r--r-- 1 sysuser sysuser 33M Jul 12 02:52 db-62881.log > 33M -rw-r--r-- 1 sysuser sysuser 33M Jul 12 02:52 db-62882.log > 33M -rw-r--r-- 1 sysuser sysuser 33M Jul 12 04:15 db-62961.log > 55M -rw-r--r-- 1 sysuser sysuser 55M Jul 14 12:18 db-63281.log > 51M -rw-r--r-- 1 sysuser sysuser 51M Jul 14 12:19 db-63282.log > 57M -rw-r--r-- 1 sysuser sysuser 57M Jul 14 12:39 db-63283.log > 57M -rw-r--r-- 1 sysuser sysuser 57M Jul 14 13:00 db-63284.log > 53M -rw-r--r-- 1 sysuser sysuser 53M Jul 14 13:31 db-63286.log > 46M -rw-r--r-- 1 sysuser sysuser 46M Jul 14 13:31 db-63287.log > 54M -rw-r--r-- 1 sysuser sysuser 54M Jul 14 13:37 db-63288.log > 35M -rw-r--r-- 1 sysuser sysuser 35M Jul 14 14:03 db-63289.log > 3.2M -rw------- 1 sysuser sysuser 3.2M Jul 14 14:14 db.redo > 113M -rw------- 1 sysuser sysuser 113M Jul 14 14:14 db.data > 29M -rw-r--r-- 1 sysuser sysuser 32M Jul 14 14:14 db-63290.log > ---------------------------------------------------------------- > Reason 62870 is not cleaned up (you can see 62869 is referenced): > 2013-07-14 14:07:39,335 | TRACE | not removing data file: 62870 as > contained > ack(s) refer to referenced file: [62713, 62800, 62801, 62802, 62803, 62804, > 62791, 62481, 62869, 62809, 62794, 62870, 62797, 62799] | > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal > Checkpoint Worker > ---------------------------------------------------------------- > I restarted ActiveMQ as we have seen that files get cleaned up on restart. > Now db-62870.log has been removed: > 33M -rw-r--r-- 1 sysuser sysuser 33M Jul 9 17:12 db-62483.log > 44M -rw-r--r-- 1 sysuser sysuser 44M Jul 9 17:16 db-62484.log > 56M -rw-r--r-- 1 sysuser sysuser 56M Jul 9 17:20 db-62485.log > 36M -rw-r--r-- 1 sysuser sysuser 36M Jul 9 17:34 db-62486.log > 33M -rw-r--r-- 1 sysuser sysuser 33M Jul 12 02:47 db-62869.log > 55M -rw-r--r-- 1 sysuser sysuser 55M Jul 14 12:18 db-63281.log > 57M -rw-r--r-- 1 sysuser sysuser 57M Jul 14 12:39 db-63283.log > 53M -rw-r--r-- 1 sysuser sysuser 53M Jul 14 13:31 db-63286.log > 54M -rw-r--r-- 1 sysuser sysuser 54M Jul 14 13:37 db-63288.log > 35M -rw-r--r-- 1 sysuser sysuser 35M Jul 14 14:03 db-63289.log > 0 -rw-r--r-- 1 sysuser sysuser 0 Jul 14 14:31 lock > 3.2M -rw------- 1 sysuser sysuser 3.2M Jul 14 14:36 db.redo > 113M -rw------- 1 sysuser sysuser 113M Jul 14 14:36 db.data > 3.0M -rw-r--r-- 1 sysuser sysuser 32M Jul 14 14:36 db-63291.log > ---------------------------------------------------------------- > This shows up in the activemq.log file: > 2013-07-14 14:32:14,245 | TRACE | gc candidates: [62870, 62881, 62882, > 62961, 63282, 63284, 63287, 63290] | > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal > Checkpoint Worker > 2013-07-14 14:32:14,245 | DEBUG | Cleanup removing the data files: [62870, > 62881, 62882, 62961, 63282, 63284, 63287, 63290] | > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal > Checkpoint Worker > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-1-Kaha-File-Cleanup-Issue-tp4669615.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
