[
https://issues.apache.org/jira/browse/AMQ-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Protasio updated AMQ-7221:
-------------------------------
Description:
Hi,
I tried to delete all my scheduled messages and I noticed that the broker stop
responding and a huge transaction file was created.
The Scheduled.db file had ~150mb and the transaction was already 140GB when I
give up and wiped the broker.
Looking at the code, I noticed that the cause of it is mostly this 2 lines:
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L512]
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L514]
And
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L501]
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L481]
On the first case, we are updating an List<Int> inside a transaction but in
order to do that, we get its value from the BTree and put back (with an added
element). When we do that, the whole list is again serialized and written
inside the transaction.
Similar thing is happening on the second case, We are decrementing one by a
number of references to a journal file but we could do it only one time in the
end saving lots of writes.
After the proposed patch, the transaction that was before more than 140GB (and
it didn't finish) was reduced to ~500 mb, and the broker could recover in less
than 1 minute (i gave up before with ~15 minutes).
On the modified test in the PR we can see that the transaction before the
change is ~50mb and after the change is ~500k
Before:
2019-06-03 23:58:19,931
[//localhost#1-1|https://issues.apache.org/localhost#1-1] - DEBUG Transaction -
Committing transaction 5177: Size 53320 kb
After:
2019-06-03 23:59:13,578
[//localhost#1-2|https://issues.apache.org/localhost#1-2] - DEBUG Transaction -
Committing transaction 5178: Size 496 kb
PS: Keeping removedJobFileIds in memory will not increase the memory usage bc
we were loading it in memory anyway when reading from the index.
was:
Hi,
I tried to delete all my scheduled messages and I noticed that the broker stop
responding and a huge transaction file was created.
The Scheduled.db file had ~150mb and the transaction was already 140GB when I
give up and wiped the broker.
Looking at the code, I noticed that the cause of it is mostly this 2 lines:
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L512]
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L514]
And
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L501]
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L481]
On the first case, we are updating an List<Int> inside a transaction but in
order to do that, we get its value from the BTree and put back (with an added
element). When we do that, the whole list is again serialized and written
inside the transaction.
Similar thing is happening on the second case, We are decrementing one by a
number of references to a journal file but we could do it only one time in the
end saving lots of writes.
After the proposed patch, the transaction that was before more than 140GB (and
it didn't finish) was reduced to ~500 mb, and the broker could recover in less
than 1 minute (i gave up before with ~15 minutes).
On the modified test in the PR we can see that the transaction before the
change is ~50mb and after the change is ~500k
Before:
2019-06-03 23:58:19,931 [//localhost#1-1] - DEBUG Transaction - Committing
transaction 5177: Size 53320 kb
After:
2019-06-03 23:59:13,578 [//localhost#1-2] - DEBUG Transaction - Committing
transaction 5178: Size 496 kb
> Delete Scheduled messages causes ActiveMQ create/write a unnecessary huge
> transaction file
> ------------------------------------------------------------------------------------------
>
> Key: AMQ-7221
> URL: https://issues.apache.org/jira/browse/AMQ-7221
> Project: ActiveMQ
> Issue Type: Improvement
> Components: KahaDB
> Reporter: Alan Protasio
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hi,
> I tried to delete all my scheduled messages and I noticed that the broker
> stop responding and a huge transaction file was created.
> The Scheduled.db file had ~150mb and the transaction was already 140GB when I
> give up and wiped the broker.
> Looking at the code, I noticed that the cause of it is mostly this 2 lines:
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L512]
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L514]
> And
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerImpl.java#L501]
> [https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/scheduler/JobSchedulerStoreImpl.java#L481]
> On the first case, we are updating an List<Int> inside a transaction but in
> order to do that, we get its value from the BTree and put back (with an added
> element). When we do that, the whole list is again serialized and written
> inside the transaction.
> Similar thing is happening on the second case, We are decrementing one by a
> number of references to a journal file but we could do it only one time in
> the end saving lots of writes.
> After the proposed patch, the transaction that was before more than 140GB
> (and it didn't finish) was reduced to ~500 mb, and the broker could recover
> in less than 1 minute (i gave up before with ~15 minutes).
> On the modified test in the PR we can see that the transaction before the
> change is ~50mb and after the change is ~500k
> Before:
> 2019-06-03 23:58:19,931
> [//localhost#1-1|https://issues.apache.org/localhost#1-1] - DEBUG Transaction
> - Committing transaction 5177: Size 53320 kb
> After:
> 2019-06-03 23:59:13,578
> [//localhost#1-2|https://issues.apache.org/localhost#1-2] - DEBUG Transaction
> - Committing transaction 5178: Size 496 kb
>
> PS: Keeping removedJobFileIds in memory will not increase the memory usage bc
> we were loading it in memory anyway when reading from the index.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
