Hi Gordon & everyone else, I finally managed to spent some quality time with the queue paging today. I have quite a lot of questions / comments / feedback. I hope you don't mind. If you want, I can as usually rise the JIRAs or provide more details for reproducing of the problems. Just let me know what is a real problem and a JIRA should be raised and what "works as designed" and no JIRA is needed.
Thanks & Regards Jakub --- 1) Why is the page file created as executable? -rwx------. 1 qpidd qpidd 110592000 Nov 6 19:07 my-test-queue-2 --- 2) As I understood from the previous discussion, the queue can store only messages which are not bigger than the page size (i.e. page_factor * memory page). Does the value of the page_factor have also some other implications? Does it have some effect on performance? --- 3) Can I somewhere see some statistics about how many pages are currently in memory or how many are off-loaded to the disk? The qpid-tool doesn't seem to have this information. --- 4) It would be really great to have a possibility to change the qpid.max_pages_loaded parameter without deleting and recreating the queue. Even having this as an "offline" option - i.e. when the change would be possible only when the broker is switched ogg - would be IMHO useful. --- 5) It would be great to have a broker configuration option "--paging-dir" to be able to specify the directory for the page files should be created. Right now it seems to use the "--data-dir" parameter. --- 6) Even when using the paging, the broker memory consumption seems to increase all the time. Is it just some sort of "index" linking to the offloaded pages? If it is only the index what is kept in memory, does that mean that with higher qpid.page_factor I will have lower memory consumption? --- 7) The purge QMF method doesn't seem to work. When I call it for the first time, it seem to always purge only the messages from the loaded pages. When I try to call it again I get the following error: Connection closed by peer with amqp:internal-error: Queue "my-test-queue-2": Dequeuing message with null persistence Id. (/home/jakub/qpid/qpid-trunk/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1372) (the queue configuration: durable=true, max_pages_loaded=1, page_factor=1). In fact with the queue configured as above, this happens to receiver as well - when reading the queue for the first time it gets only the messages from the loaded page. When read for the second time, it creates the same error. I also tried to change the values of max_pages_loaded and page_factor ... but that doesn't seem to change much. I can read/purge more messages before seeing the error. But it still seems to occur regularly. --- 8) When the queue is created as above (max_pages_loaded=1, page_factor=1), but as non-durable, purging and reading still doesn't work properly. It always reads / purges only the loaded pages. But of course I do not get the error from the persistent message store. However, with increase max_pages_loaded and/or page_factor (e.g. 10 and 10), the problem seems to be resolved (for non-persistent queues). It seems to me that when the pages are too small / only few pages are loaded, the broker doesn't manage to load the pages quickly enough from disk and that some kind of "starves" the receiver. Perhaps setting minimal values for the page_factor / max_pages_loaded can help here? --- 9) I know that the paged queues don't support LVQ functionality and probably also message priorities. But it seems that paged queues don't really support filters / selectors as well, or? The client using selector always receives only the messages which match the selector and are in the loaded pages. I guess it makes sense - since the broker doesn't keep complete message headers, it cannot easily match the selectors. But in that case, I would suggest to raise an error if a consumer with filter/selector is created. --- 10) When I try to send a message bigger than the page size, I get following error: Link detached by peer with amqp:precondition-failed: precondition-failed: Message is larger than page size for queue backed by /home/qpidd/qpid/0000/data/broker/my-test-queue-2 (/home/jakub/qpid/qpid-trunk/cpp/src/qpid/broker/PagedQueue.cpp:107) Is it intentional to include the full path to the page file? I would definitely prefer to show just the file name or a queue name. ---
