[
https://issues.apache.org/jira/browse/QPID-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16858805#comment-16858805
]
Kim van der Riet commented on QPID-8320:
----------------------------------------
commit 5d549d8409f19707f68ca0fbf24ee4e0d3ae8de6 (HEAD -> master, origin/master,
origin/HEAD)
Author: Kim van der Riet <[email protected]>
Date: Fri Jun 7 12:39:40 2019 -0400
QPID-8320: Fix for empty journal file leak when linearstore recovers
> [linearstore] Empty journal files orphaned and accumulate when the broker is
> restarted
> --------------------------------------------------------------------------------------
>
> Key: QPID-8320
> URL: https://issues.apache.org/jira/browse/QPID-8320
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Reporter: Kim van der Riet
> Assignee: Kim van der Riet
> Priority: Major
>
> If a queue journal is filled until the last file in the journal is full, then
> the store preemptively adds a new journal file to the queue store. This new
> file is at first uninitialized, but is initialized when it is first used.
> If, at recovery, such an uninitialized file exists in a journal, then on
> recovery, this file is ignored, and a new uninitialized file is added. Hence
> two uninitialized files now exist in the journal. If the broker is repeatedly
> stopped, then started with a journal in this state, a new uninitialized file
> is added for each restart.
> In addition, the journal recovery does not dispose of the unused
> uninitialized files, so they accumulate and continue to exist through
> multiple restarts.
> +*Reproducer:*+
> Start with a clean store:
> {noformat}
> rm -rf ~/.qpidd
> {noformat}
> Start the broker, then:
> {noformat}
> $ qpid-config add queue --durable test_queue
> $ ls ~/.qpidd/qls/jrnl2/test_queue/
> f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
> $ hexdump -C
> ~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
> 00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
> 00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
> 00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00201000
> {noformat}
>
> which is an uninitialized empty journal file. Now add 1024 messages that
> when encoded consume almost 2048 bytes per message on disk. This should fill
> the first file exactly, so that the last enqueue record coincides with the
> physical end of the file at offset {{0x201000}}:
> {noformat}
> $ qpid-send -a test_queue --durable=yes -m 1024 --content-size=1865
> $ ls ~/.qpidd/qls/jrnl2/test_queue/
> e404051f-8af7-422d-a088-7e957c4db3af.jrnl
> f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
> $ hexdump -C
> ~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl |
> grep QLS
> 00000000 51 4c 53 66 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSf....O.s ....|
> 00001000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> 00001800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> ...
> 001ff800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> 00200000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> 00200800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> {noformat}
> Check that the newly added file is empty:
> {noformat}
> hexdump -C
> ~/.qpidd/qls/jrnl2/test_queue/e404051f-8af7-422d-a088-7e957c4db3af.jrnl
> 00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
> 00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
> 00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00201000
> {noformat}
> It is important to check that the second file is empty other than the file
> header. If there are any records present, then the file will not be
> considered empty during recovery, and the conditions for the bug will not be
> met. Depending on network and threading conditions, the store may add in
> filler records {{"QLSx"}} at one or two points during the writing of the
> files, so this may push the final record to be written in the second file. If
> this happens, try again, or adjust the number of records down slightly until
> this condition is met.
> Once this condition has been met, stop the broker, then restart it. There
> will now be two empty files present, the original, plus a new one added at
> the broker restart.
> Start and stop the broker several times. For each recovery, one new empty
> file is added to the journal. The old files are still present, but are
> orphaned, and are never moved, used or put back into the Empty File Pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
