This problem can occur if the messages are consumed out-of-order, or
more specifically, if there exists and "orphaned" message which has been
enqueued some time back, but not consumed.
The linearstore assumes that all messages are consumed (more-or-less) in
the order they are sent to the broker. Files are freed from the
linearstore as the "leading" files become freed of all stored messages.
If a message should be enqueued but not consumed, then the journal
file containing that message will not be freed, and the files following
that file will also be blocked from being freed (as this would
compromise the ability to recover the state of the store if there should
be a stoppage).
If your messaging scenario does not meet the in-order criterion, and
message consumption is random with possible long periods of storage for
some messages, then it is possible that linearstore will not be a
suitable store. A store with random read/write (possibly based on a
database) would be better for this use-case.
I see from your logs that a single message is enqueued. If that message
is an orphaned message then that would explain the growth in journal
files. Assuming this to be true:
Strategies for preventing this depend on your use-case:
1. The best way is to avoid orphaned messages in the first place. This
will ensure that the journal is always short and the individual journal
files are being re-used.
2. If the presence of unconsumed messages for long periods is
unavoidable and the orphaned messages are not wanted, then one way to
control this is to set a Time-to-Live on each message such that when
that expires, the message is deleted by the broker automatically, which
would have the effect of freeing up the linearstore journal files. (I
have not tested this strategy, but I believe it should work.)
3. The last and more complex way to handle this is to have a small
client find and either remove the messages (if they are unwanted), or
remove and resend the messages at regular intervals (if the messages are
wanted and must be preserved).
If the message in your log is recent, and not orphaned, then there could
be a bug in the store which needs addressing. If this is so, then it
would be helpful (if possible) to get a copy of the store files so that
I can analyze why the process of freeing them is "stuck".
There is a proposal to add a limit to the queue expansion (ie set a
maximum number of files allowed by a single queue and/or combined
queues. After the limit is exceeded, further enqueues would be refused.
I am not sure if such a feature would help your situation, as it would
prevent unlimited growth in journal files, but at the cost of having
client message publishing operations fail owing to the full store.
I hope this helps,
Kim van der Riet
On 12/09/2015 08:50 AM, Adrian Cislo wrote:
Hello,
We have been recently suffering a quite critical problem with journal
files on production environment.
After some time, even though queues seem to be empty, Qpid allocates a
lot of disk space for journal files.
Qpid is in version 0.30, rhel6 distribution. All running on RHEL 6.5
Enclosed you will find some sample logs from that environment:
9.12.15_jrnl_size.txt – results from du command
Qpid-stat-q.txt – results from qpid-stat –q
qpid_vsi_qpid-tool-result.txt, qpid_request_qpid-tool-result.txt,
qpid_public_qpid-tool-result.txt – results from qpid-tool, show #
The question is, why does it happen, and if/how we can prevent it ?
*Best regards*
**
*FARA*
*Adrian Cisło*
Software Engineer
Mobile: +48790618366**
E-mail : [email protected] <mailto:[email protected]>__
/TECHNOLOGY IN MOTION///
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
