[
https://issues.apache.org/jira/browse/ARTEMIS-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabio Nascimento Brandão updated ARTEMIS-3868:
----------------------------------------------
Description:
We are having some problems with disk space usage. Basically we have a lot of
queues and one of them fail to consume the messages (problem in a microservice
for about 10 minutes).
Then we use the message redelivery feature to process these messages again
after some delay.
The problems ocurrs with de journal compact. This process creates a lot of
files until fill the disk.
I took some created files at production server and tried to compact them. The
disk usage grow a lot and only after the second compact process the disk usage
stayed at an acceptable level.
I attached a "test" to reproduce the problem. The test create one journal file
with 2048 records and using only 2,1MB. The records have an information that is
the "compact count". I adjusted this field to 1 and 2.
What I found out is when the compactor process a record with this field less
than 2 (after some record being greater or equals to 2), the JournalCompactor
creates a new file.
In this test, when I start the journal it creates a new file of 10MB (default
configuration) and keeps the 2,1MB file.
When I ran the compact process, It creates 1024 files and keeps the old 2. 1024
* 10MB = 10GB!!!
If I ran the compact process again, It shrink to only 2 files of 10MB!!!
The problem seems to be in the JournalCompactor at method checkCompact. I think
we can remove this method and assume the flow like it returned false:
[https://github.com/apache/activemq-artemis/blob/main/artemis-journal/src/main/java/org/apache/activemq/artemis/core/journal/impl/JournalCompactor.java#L190-L203]
This was an old code that was introduced at HornetQ:
[https://github.com/hornetq/hornetq/commit/93af1cb92f4050e54e83c8daa9c67ce43dbcfead]
I also attached an image with the disk usage of my production server with the
problem.
was:
We are having some problems with disk space usage. Basically we have a lot of
queues and one of them fail to consume the messages (problem in a microservice
for about 10 minutes).
Then we use the message redelivery feature to process these messages again
after some delay.
The problems ocurrs with de journal compact. This process creates a lot of
files until fill the disk.
I took some created files at production server and tried to compact them. The
disk usage grow a lot and only after the second compact process the disk usage
stayed at an acceptable level.
I attached a "test" to reproduce the problema. The test create one journal file
with 2048 records and using only 2,1MB. The records have an information that is
the "compact count". I adjusted this field to 1 and 2.
What I found out is when the compactor process a record with this field less
than 2 (after some record being greater or equals to 2), the JournalCompactor
creates a new file.
In this test, when I start the journal it creates a new file of 10MB (default
configuration) and keeps the 2,1MB file.
When I ran the compact process, It creates 1024 files and keeps the old 2. 1024
* 10MB = 10GB!!!
If I ran the compact process again, It shrink to only 2 files of 10MB!!!
The problem seems to be in the JournalCompactor at method checkCompact. I think
we can remove this method and assume the flow like it returned false:
[https://github.com/apache/activemq-artemis/blob/main/artemis-journal/src/main/java/org/apache/activemq/artemis/core/journal/impl/JournalCompactor.java#L190-L203]
This was an old code that was introduced at HornetQ:
[https://github.com/hornetq/hornetq/commit/93af1cb92f4050e54e83c8daa9c67ce43dbcfead]
I also attached an image with the disk usage of my production server with the
problem.
> Compact process filling all the disk
> ------------------------------------
>
> Key: ARTEMIS-3868
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3868
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: ActiveMQ-Artemis-Native
> Affects Versions: 2.19.0, 2.20.0, 2.21.0, 2.22.0, 2.23.0, 2.23.1
> Reporter: Fabio Nascimento Brandão
> Assignee: Clebert Suconic
> Priority: Major
> Attachments: CompactDiskUsage.java, image-2022-06-21-19-33-59-276.png
>
>
> We are having some problems with disk space usage. Basically we have a lot of
> queues and one of them fail to consume the messages (problem in a
> microservice for about 10 minutes).
> Then we use the message redelivery feature to process these messages again
> after some delay.
> The problems ocurrs with de journal compact. This process creates a lot of
> files until fill the disk.
> I took some created files at production server and tried to compact them. The
> disk usage grow a lot and only after the second compact process the disk
> usage stayed at an acceptable level.
> I attached a "test" to reproduce the problem. The test create one journal
> file with 2048 records and using only 2,1MB. The records have an information
> that is the "compact count". I adjusted this field to 1 and 2.
> What I found out is when the compactor process a record with this field less
> than 2 (after some record being greater or equals to 2), the JournalCompactor
> creates a new file.
> In this test, when I start the journal it creates a new file of 10MB (default
> configuration) and keeps the 2,1MB file.
> When I ran the compact process, It creates 1024 files and keeps the old 2.
> 1024 * 10MB = 10GB!!!
> If I ran the compact process again, It shrink to only 2 files of 10MB!!!
> The problem seems to be in the JournalCompactor at method checkCompact. I
> think we can remove this method and assume the flow like it returned false:
> [https://github.com/apache/activemq-artemis/blob/main/artemis-journal/src/main/java/org/apache/activemq/artemis/core/journal/impl/JournalCompactor.java#L190-L203]
> This was an old code that was introduced at HornetQ:
> [https://github.com/hornetq/hornetq/commit/93af1cb92f4050e54e83c8daa9c67ce43dbcfead]
> I also attached an image with the disk usage of my production server with the
> problem.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
