Ryan Senior created AMQ-6152:
--------------------------------
Summary: KahaDB scheduler log files not being deleted
Key: AMQ-6152
URL: https://issues.apache.org/jira/browse/AMQ-6152
Project: ActiveMQ
Issue Type: Bug
Components: KahaDB
Affects Versions: 5.13.0, 5.12.0, 5.11.1
Environment: Linux (issue reported on several versions)
OpenJDK 1.8.0_72
Reporter: Ryan Senior
Attachments: BasicSchedulerTest.java
Something is holding onto KahaDB scheduler log files. We have reports of up to
400GB of scheduler log files. I have tried to isolate the issue and create a
minimal example (attached). In the troubleshooting I have done, the scheduler
GC process is running, it's just deciding not to GC files that it should be. I
have also found behavior inconsistent on the log files it does remove.
The ran the attached example/test on 5.10.0, 5.11.1, 5.12.0 and 5.13.0. The
test schedules 20 messages that are large enough to cause 4 log files to be
created. It then consumes all 20 messages. When on 5.10.0, it behaves like I
would expect, files 1-3 are GC'd and the 4th (the current log file) is left. On
all other versions I've tried it always leaves the first 2 files, and sometimes
will GC the 3rd.
Below is a snippet from the log of the scheduler process and why it's deciding
not to GC these files:
{noformat}
2016-02-01 16:32:33,327 DEBUG [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Job
Scheduler Store Checkpoint started.
2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Last
update: 4:29447367, full gc candidates set: [1, 2, 3, 4]
2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] gc
candidates after reference check: [1, 2, 3]
2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A
remove in log 1 has an add still in existance.
2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A
remove in log 2 has an add still in existance.
2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A
remove in log 3 has an add still in existance.
2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] gc
candidates after removals check: []
2016-02-01 16:32:33,339 DEBUG [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Job
Scheduler Store Checkpoint complete.
2016-02-01 16:32:33,727 INFO [main] [o.a.a.s.k.s.JobSchedulerStoreImpl]
JobSchedulerStore: test-mq/localhost/scheduler stopped.
{noformat}
This issue was originally reported in the Open Source PuppetDB project, ticket
[here|https://tickets.puppetlabs.com/browse/PDB-1411].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
