Hello, Thanks for reply :)
I'd like to continue following thread http://mail-archives.apache.org/mod_mbox/qpid-dev/201512.mbox/%[email protected]%3E >> The linearstore assumes that all messages are consumed (more-or-less) in the order they are sent to the broker. In our implementation each queue has its own consumer, there is always one consumer per one queue. Every consumer reads, processes and acks message in the broker, and then read next message, processes it and acks, and so on. So they are read (in theory) in a row. Of course per queue. We tested it again but using new Qpid 0.34 (built from sources). Similar situation happened, but it gets back to the normal state after restarting the broker. For example, during the weekend we ran some performance test against qpid. We noticed that even though queues seemed to be empty, journal files were huge. [root@localhost ~]# ls -la /var/lib/qpidd/qls/jrnl2/no.fara.cs.eventing.queues.vsi-integration-module/ total 32 drwxr-x--- 2 qpidd qpidd 4096 Jan 10 19:39 . drwxr-x--- 42 qpidd qpidd 4096 Jan 5 15:46 .. lrwxrwxrwx 1 qpidd qpidd 83 Jan 9 02:03 21948ff2-8d19-4dd8-b2cd-49cb6c63a842.jrnl -> /var/lib/qpidd/qls/p001/efp/10000k/in_use/21948ff2-8d19-4dd8-b2cd-49cb6c63a842.jrnl lrwxrwxrwx 1 qpidd qpidd 83 Jan 5 15:46 3604c9d0-f43e-4868-9198-1244c608a37a.jrnl -> /var/lib/qpidd/qls/p001/efp/10000k/in_use/3604c9d0-f43e-4868-9198-1244c608a37a.jrnl lrwxrwxrwx 1 qpidd qpidd 83 Jan 7 10:05 3a3b5af2-5e18-4fc5-8d78-6cf8e8ecddd0.jrnl -> /var/lib/qpidd/qls/p001/efp/10000k/in_use/3a3b5af2-5e18-4fc5-8d78-6cf8e8ecddd0.jrnl lrwxrwxrwx 1 qpidd qpidd 83 Jan 9 22:51 7547f08f-5cf4-49dd-8710-d28caf0deb52.jrnl -> /var/lib/qpidd/qls/p001/efp/10000k/in_use/7547f08f-5cf4-49dd-8710-d28caf0deb52.jrnl lrwxrwxrwx 1 qpidd qpidd 83 Jan 10 19:39 937bebf6-b455-4191-9e7a-d0f43fa3304e.jrnl -> /var/lib/qpidd/qls/p001/efp/10000k/in_use/937bebf6-b455-4191-9e7a-d0f43fa3304e.jrnl lrwxrwxrwx 1 qpidd qpidd 83 Jan 8 09:18 a4baf288-5c43-44dd-8dd2-7591ac5f6b5d.jrnl -> /var/lib/qpidd/qls/p001/efp/10000k/in_use/a4baf288-5c43-44dd-8dd2-7591ac5f6b5d.jrnl We restarted Qpid and saw in logs: Jan 11 08:23:02 localhost qpidd[31069]: 2016-01-11 08:23:02 [Store] notice Linear Store: Journal "no.fara.cs.eventing.queues.vsi-integration-module": Files removed from head of journal: prior truncation during recovery:#012 /var/lib/qpidd/qls/jrnl2/no.fara.cs.eventing.queues.vsi-integration-module/937bebf6-b455-4191-9e7a-d0f43fa3304e.jrnl#012 /var/lib/qpidd/qls/jrnl2/no.fara.cs.eventing.queues.vsi-integration-module/7547f08f-5cf4-49dd-8710-d28caf0deb52.jrnl#012 /var/lib/qpidd/qls/jrnl2/no.fara.cs.eventing.queues.vsi-integration-module/21948ff2-8d19-4dd8-b2cd-49cb6c63a842.jrnl#012 /var/lib/qpidd/qls/jrnl2/no.fara.cs.eventing.queues.vsi-integration-module/a4baf288-5c43-44dd-8dd2-7591ac5f6b5d.jrnl#012 * /var/lib/qpidd/qls/jrnl2/no.fara.cs.eventing.queues.vsi-integration-module/3a3b5af2-5e18-4fc5-8d78-6cf8e8ecddd0.jrnl After that we noticed that journal files' structure went back to the normal shape. We're going to restart Qpid every day and treat it as some temporary workaround. Is it possible to configure qpid that this 'recovery' is done on the fly (without restarting) ? Also we will set up TTL for every message, now it's configured for some 'categories' only. If you need store files, probably we can deliver them somehow. Actually it's a huge package now and I wonder how we can send them to you. From Kim van der Riet <[email protected]> Subject Re: Qpid-cpp-server-linearstore problem with growing up files infinitely. Date Thu, 10 Dec 2015 17:00:16 GMT This problem can occur if the messages are consumed out-of-order, or more specifically, if there exists and "orphaned" message which has been enqueued some time back, but not consumed. The linearstore assumes that all messages are consumed (more-or-less) in the order they are sent to the broker. Files are freed from the linearstore as the "leading" files become freed of all stored messages. If a message should be enqueued but not consumed, then the journal file containing that message will not be freed, and the files following that file will also be blocked from being freed (as this would compromise the ability to recover the state of the store if there should be a stoppage). If your messaging scenario does not meet the in-order criterion, and message consumption is random with possible long periods of storage for some messages, then it is possible that linearstore will not be a suitable store. A store with random read/write (possibly based on a database) would be better for this use-case. I see from your logs that a single message is enqueued. If that message is an orphaned message then that would explain the growth in journal files. Assuming this to be true: Strategies for preventing this depend on your use-case: 1. The best way is to avoid orphaned messages in the first place. This will ensure that the journal is always short and the individual journal files are being re-used. 2. If the presence of unconsumed messages for long periods is unavoidable and the orphaned messages are not wanted, then one way to control this is to set a Time-to-Live on each message such that when that expires, the message is deleted by the broker automatically, which would have the effect of freeing up the linearstore journal files. (I have not tested this strategy, but I believe it should work.) 3. The last and more complex way to handle this is to have a small client find and either remove the messages (if they are unwanted), or remove and resend the messages at regular intervals (if the messages are wanted and must be preserved). If the message in your log is recent, and not orphaned, then there could be a bug in the store which needs addressing. If this is so, then it would be helpful (if possible) to get a copy of the store files so that I can analyze why the process of freeing them is "stuck". There is a proposal to add a limit to the queue expansion (ie set a maximum number of files allowed by a single queue and/or combined queues. After the limit is exceeded, further enqueues would be refused. I am not sure if such a feature would help your situation, as it would prevent unlimited growth in journal files, but at the cost of having client message publishing operations fail owing to the full store. I hope this helps, Kim van der Riet From: Adrian Cislo Sent: Wednesday, December 09, 2015 2:50 PM To: '[email protected]' <[email protected]> Subject: Qpid-cpp-server-linearstore problem with growing up files infinitely. Hello, We have been recently suffering a quite critical problem with journal files on production environment. After some time, even though queues seem to be empty, Qpid allocates a lot of disk space for journal files. Qpid is in version 0.30, rhel6 distribution. All running on RHEL 6.5 Enclosed you will find some sample logs from that environment: 9.12.15_jrnl_size.txt - results from du command Qpid-stat-q.txt - results from qpid-stat -q qpid_vsi_qpid-tool-result.txt, qpid_request_qpid-tool-result.txt, qpid_public_qpid-tool-result.txt - results from qpid-tool, show # The question is, why does it happen, and if/how we can prevent it ? Best regards FARA Adrian Cisło Software Engineer Mobile: +48790618366 E-mail : [email protected]<mailto:[email protected]> TECHNOLOGY IN MOTION
