[
https://issues.apache.org/jira/browse/QPID-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kim van der Riet updated QPID-8320:
-----------------------------------
Description:
If a queue journal is filled until the last file in the journal is full, then
the store preemptively adds a new journal file to the queue store. This new
file is at first uninitialized, but is initialized when it is first used.
If, at recovery, such an uninitialized file exists in a journal, then on
recovery, this file is ignored, and a new uninitialized file is added. Hence
two uninitialized files now exist in the journal. If the broker is repeatedly
stopped, then started with a journal in this state, a new uninitialized file is
added for each restart.
In addition, the journal recovery does not dispose of the unused uninitialized
files, so they accumulate and continue to exist through multiple restarts.
+*Reproducer:*+
Start with a clean store:
{noformat}
rm -rf ~/.qpidd
{noformat}
Start the broker, then:
{noformat}
$ qpid-config add queue --durable test_queue
$ ls ~/.qpidd/qls/jrnl2/test_queue/
f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
$ hexdump -C
~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00201000
{noformat}
which is an uninitialized empty journal file. Now add 1024 messages that when
encoded consume almost 2048 bytes per message on disk. This should fill the
first file exactly, so that the last enqueue record coincides with the physical
end of the file at offset {{0x201000}}:
{noformat}
$ qpid-send -a test-queue --durable=yes -m 1024 --content-size=1865
$ ls ~/.qpidd/qls/jrnl2/test_queue/
e404051f-8af7-422d-a088-7e957c4db3af.jrnl
f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
$ hexdump -C
~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl | grep
QLS
00000000 51 4c 53 66 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSf....O.s ....|
00001000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
00001800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
...
001ff800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
00200000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
00200800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
{noformat}
Check that the newly added file is empty:
{noformat}
hexdump -C
~/.qpidd/qls/jrnl2/test_queue/e404051f-8af7-422d-a088-7e957c4db3af.jrnl
00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00201000
{noformat}
It is important to check that the second file is empty other than the file
header. If there are any records present, then the file will not be considered
empty during recovery, and the conditions for the bug will not be met.
Depending on network and threading conditions, the store may add in filler
records {{"QLSx"}} at one or two points during the writing of the files, so
this may push the final record to be written in the second file. If this
happens, try again, or adjust the number of records down slightly until this
condition is met.
Once this condition has been met, stop the broker, then restart it. There will
now be two empty files present, the original, plus a new one added at the
broker restart.
Start and stop the broker several times. For each recovery, one new empty file
is added to the journal. The old files are still present, but are orphaned, and
are never moved, used or put back into the Empty File Pool.
was:
If a queue journal is filled until the last file in the journal is full, then
the store preemptively adds a new journal file to the queue store. This new
file is at first uninitialized, but is initialized when it is first used.
If, at recovery, such an uninitialized file exists in a journal, then on
recovery, this file is ignored, and a new uninitialized file is added. Hence
two uninitialized files now exist in the journal. If the broker is repeatedly
stopped, then started with a journal in this state, a new uninitialized file is
added for each restart.
In addition, the journal recovery does not dispose of the unused uninitialized
files, so they accumulate and continue to exist through multiple restarts.
+*Reproducer:*+
Start with a clean store:
{noformat}
rm -rf ~/.qpidd
{noformat}
Start the broker, then:
{noformat}
$ qpid-config add queue --durable test_queue
$ ls ~/.qpidd/qls/jrnl2/test_queue/
f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
$ hexdump -C
~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00201000
{noformat}
which is an uninitialized empty journal file. Now add 1024 messages that when
encoded consume almost 2048 bytes per message on disk. This should fill the
first file exactly, so that the last enqueue record coincides with the physical
end of the file at offset {{0x201000}}:
{noformat}
$ qpid-send -a test-queue --durable=yes -m 1024 --content-size=1865
$ ls ~/.qpidd/qls/jrnl2/test_queue/
e404051f-8af7-422d-a088-7e957c4db3af.jrnl
f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
$ hexdump -C
~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl | grep
QLS
00000000 51 4c 53 66 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSf....O.s ....|
00001000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
00001800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
...
001ff800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
00200000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
00200800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
{noformat}
Check that the newly added file is empty:
{noformat}
hexdump -C
~/.qpidd/qls/jrnl2/test_queue/e404051f-8af7-422d-a088-7e957c4db3af.jrnl
00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00201000
{noformat}
It is important to check that the second file is empty other than the file
header. If there are any records present, then the file will not be considered
empty during recovery, and the conditions for the bug will not be met.
Depending on network conditions, the store may add in filler records {{"QLSx"}}
at one or two points during the writing of the files, so this may push the
final record to be written in the second file. If this happens, try again, or
adjust the number of records down slightly until this condition is met.
Once this condition has been met, stop the broker, then restart it. There will
now be two empty files present, the original, plus a new one added at the
broker restart.
Start and stop the broker several times. For each recovery, one new empty file
is added to the journal. The old files are still present, but are orphaned, and
are never moved, used or put back into the Empty File Pool.
> [linearstore] Empty journal files orphaned and accumulate when the broker is
> restarted
> --------------------------------------------------------------------------------------
>
> Key: QPID-8320
> URL: https://issues.apache.org/jira/browse/QPID-8320
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Reporter: Kim van der Riet
> Assignee: Kim van der Riet
> Priority: Major
>
> If a queue journal is filled until the last file in the journal is full, then
> the store preemptively adds a new journal file to the queue store. This new
> file is at first uninitialized, but is initialized when it is first used.
> If, at recovery, such an uninitialized file exists in a journal, then on
> recovery, this file is ignored, and a new uninitialized file is added. Hence
> two uninitialized files now exist in the journal. If the broker is repeatedly
> stopped, then started with a journal in this state, a new uninitialized file
> is added for each restart.
> In addition, the journal recovery does not dispose of the unused
> uninitialized files, so they accumulate and continue to exist through
> multiple restarts.
> +*Reproducer:*+
> Start with a clean store:
> {noformat}
> rm -rf ~/.qpidd
> {noformat}
> Start the broker, then:
> {noformat}
> $ qpid-config add queue --durable test_queue
> $ ls ~/.qpidd/qls/jrnl2/test_queue/
> f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
> $ hexdump -C
> ~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
> 00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
> 00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
> 00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00201000
> {noformat}
>
> which is an uninitialized empty journal file. Now add 1024 messages that
> when encoded consume almost 2048 bytes per message on disk. This should fill
> the first file exactly, so that the last enqueue record coincides with the
> physical end of the file at offset {{0x201000}}:
> {noformat}
> $ qpid-send -a test-queue --durable=yes -m 1024 --content-size=1865
> $ ls ~/.qpidd/qls/jrnl2/test_queue/
> e404051f-8af7-422d-a088-7e957c4db3af.jrnl
> f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl
> $ hexdump -C
> ~/.qpidd/qls/jrnl2/test_queue/f965476e-eea0-4c02-be50-cbfbce6da71a.jrnl |
> grep QLS
> 00000000 51 4c 53 66 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSf....O.s ....|
> 00001000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> 00001800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> ...
> 001ff800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> 00200000 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> 00200800 51 4c 53 65 02 00 00 00 4f c8 73 20 db c2 df 1d |QLSe....O.s ....|
> {noformat}
> Check that the newly added file is empty:
> {noformat}
> hexdump -C
> ~/.qpidd/qls/jrnl2/test_queue/e404051f-8af7-422d-a088-7e957c4db3af.jrnl
> 00000000 51 4c 53 66 02 00 00 00 00 00 00 00 00 00 00 00 |QLSf............|
> 00000010 00 00 00 00 00 00 00 00 01 00 01 00 00 00 00 00 |................|
> 00000020 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
> *
> 00201000
> {noformat}
> It is important to check that the second file is empty other than the file
> header. If there are any records present, then the file will not be
> considered empty during recovery, and the conditions for the bug will not be
> met. Depending on network and threading conditions, the store may add in
> filler records {{"QLSx"}} at one or two points during the writing of the
> files, so this may push the final record to be written in the second file. If
> this happens, try again, or adjust the number of records down slightly until
> this condition is met.
> Once this condition has been met, stop the broker, then restart it. There
> will now be two empty files present, the original, plus a new one added at
> the broker restart.
> Start and stop the broker several times. For each recovery, one new empty
> file is added to the journal. The old files are still present, but are
> orphaned, and are never moved, used or put back into the Empty File Pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
