Kim, Any tools to read or dump messages from journal files?
Thanks, Ram On Fri, Dec 7, 2018 at 11:32 AM rammohan ganapavarapu < [email protected]> wrote: > Kim, > > We have only one main queue and one dead letter queue, and we have 10 > produces and 12 consumers and producers pump 1k messages/sec. Below is > qpid-stat -q output when it stopped taking any more messages. > > bash-4.1# qpid-stat -q > Queues > queue dur autoDel excl msg > msgIn msgOut bytes bytesIn bytesOut cons bind > > ========================================================================================================================= > ax-q-axgroup-001-consumer-group-001 Y 0 > 114k 114k 0 5.88g 5.88g 11 2 > ax-q-axgroup-001-consumer-group-001-dl Y 0 > 0 0 0 0 0 0 2 > d72e183e-f0df-457c-89a7-81a2cff509c8:0.0 Y Y 0 > 0 0 0 0 0 1 2 > > > Thanks, > Ram > > > > > On Fri, Dec 7, 2018 at 9:00 AM Kim van der Riet <[email protected]> > wrote: > >> This looks like a bug to me, and that is why I am keen to see a >> reproducer if you can find one. How many queues are there? How many >> producers and consumers are there for each queue? How are the consumers >> working? Are they configured as listeners, or do they poll for new >> messages? How frequently? How long does it take under these conditions >> for the error to occur typically? If I can get some kind of idea what >> the runtime conditions are, it will give me some idea where to look. >> >> If you set the broker to use INFO+ logging (log-enable=info+), then you >> should see some detail about the starting and recovery of the store when >> the broker starts, which should include this info. The store settings in >> the config file are global, so when you set a particular buffer >> configuration, all queues will use this. It should be reported during >> startup when using INFO+ level logging. Watch your log size, however, as >> using this level will make the logs big. >> >> On 12/5/18 5:06 PM, rammohan ganapavarapu wrote: >> > Kim, >> > >> > We have set wcache-page-size=128 in qpidd.conf, restarted broker and let >> > client recreated the queues fresh, we still getting this error, how do >> we >> > find if queues created by client actually have this >> wcache-page-size=128? >> > >> > 2018-12-05 21:18:16 [Protocol] error Connection >> > qpid.<server>:5672-<client>:17769 closed by error: Queue <queue-name>: >> > MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() >> threw >> > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned >> > partly completed (state ENQ_PART). (This data_tok: id=456535 state=NONE) >> > >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) >> > >> > Thanks, >> > Ram >> > >> > >> > >> > On Tue, Dec 4, 2018 at 8:18 AM rammohan ganapavarapu < >> > [email protected]> wrote: >> > >> >> Kim, >> >> >> >> Thank you, i will play with that setting, please let me know if any >> other >> >> tunings will help. >> >> >> >> Ram >> >> >> >> On Wed, Nov 28, 2018 at 8:04 AM Kim van der Riet <[email protected]> >> >> wrote: >> >> >> >>> The answer to your first question depends on what is more important to >> >>> you - low latency or high throughput. Messages to be persisted will >> >>> accumulate in a buffer page until it is full or until a timer is >> >>> triggered, then it will be written to disk. It is not until this >> happens >> >>> that the message will be acknowledged by the broker. If low latency is >> >>> important, then having smaller but more numerous buffer pages will >> mean >> >>> the messages will not wait for very long before being written to disk >> >>> and acknowledged as received. However this occurs at the cost of some >> >>> efficiency, which can affect throughput. If you have large volumes of >> >>> messages and the throughput is more important, then using fewer but >> >>> larger buffer pages will help you. >> >>> >> >>> Be aware, however, that the product of the size and number of pages is >> >>> the total memory that will be consumed and held by the broker for >> >>> buffering *per queue*. If you have a very large number of queues, then >> >>> you must watch out that you don't over-size your write buffers or else >> >>> you will run out of memory. >> >>> >> >>> While I cannot give you specific answers, as these depend on your >> >>> performance priorities, I suggest some trial-and-error if you want to >> >>> adjust these values. >> >>> >> >>> The Transaction Prepared List (TPL) is a special global queue for >> >>> persisting transaction boundaries. As this info is usually small and >> >>> relatively infrequent, the tpl-* settings apply to this queue only and >> >>> the user has the option to use different values than the regular >> queues. >> >>> If you don't use transactions, then this can be ignored. It is not a >> >>> queue that can be written to directly, but the store creates its own >> >>> data that is saved in this queue. Adjusting the tpl-* settings depends >> >>> only on the frequency of transactions in the user's application or >> >>> use-case. >> >>> >> >>> Hope that helps, >> >>> >> >>> Kim van der Riet >> >>> >> >>> On 11/27/18 4:44 PM, rammohan ganapavarapu wrote: >> >>>> Kim, >> >>>> >> >>>> 1. My message size is around 80kb, so what would be suggested values >> for >> >>>> the blow properties? >> >>>> >> >>>> >> >>>> wcache-page-size >> >>>> wcache-num-pages >> >>>> tpl-wcache-num-pages >> >>>> tpl-wcache-page-size >> >>>> >> >>>> right now i have all defaults, so i am trying to see if i can tune >> these >> >>>> values for my messages size to avoid those AIO busy cases. I have >> try >> >>> to >> >>>> define those properties/options in qpidd.conf file but when i run >> >>>> qpid-config queues its not showing those values on my queues created >> by >> >>>> client application, do i have to define those options when i create >> >>> queue >> >>>> instead of keep them in qpidd.conf? >> >>>> >> >>>> 2. What is difference b/w tpl-wcache-page-size and wcache-page-size >> >>>> >> >>>> Thanks, >> >>>> Ram >> >>>> >> >>>> On Fri, Nov 16, 2018 at 9:26 AM Kim van der Riet < >> [email protected]> >> >>>> wrote: >> >>>> >> >>>>> There is little documentation on linearstore. Certainly, the Apache >> >>> docs >> >>>>> don't contain much. I think this is an oversight, but it won't get >> >>> fixed >> >>>>> anytime soon. >> >>>>> >> >>>>> Kim >> >>>>> >> >>>>> On 11/16/18 12:11 PM, rammohan ganapavarapu wrote: >> >>>>>> Any one point me to the doc where i can read internals about how >> >>>>>> linearstore works and how qpid uses it? >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Ram >> >>>>>> >> >>>>>> On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < >> >>>>>> [email protected]> wrote: >> >>>>>> >> >>>>>>> Kim, >> >>>>>>> >> >>>>>>> Thanks for clearing that up for me, does it support SAN storage >> >>> blocks. >> >>>>>>> Where can i read more about linearstore if i want to know the low >> >>> level >> >>>>>>> internals? >> >>>>>>> >> >>>>>>> Ram >> >>>>>>> >> >>>>>>> On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet < >> >>> [email protected]> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> The linearstore relies on using libaio for its async disk writes. >> >>> The >> >>>>>>>> O_DIRECT flag is used, and this requires a block of aligned >> memory >> >>> to >> >>>>>>>> serve as a memory buffer for disk write operations. To my >> knowledge, >> >>>>>>>> this technique only works with local disks and controllers. NFS >> does >> >>>>> not >> >>>>>>>> allow for DMA memory writes to disk AFAIK, and for as long as I >> can >> >>>>>>>> remember, has been a problem for the linearstore. With some work >> it >> >>>>>>>> might be possible to make it work using another write technique >> >>> though. >> >>>>>>>> NFS has never been a "supported" medium for linearstore. >> >>>>>>>> >> >>>>>>>> On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: >> >>>>>>>>> But how does NFS will cause this issue, i am interested to see >> >>> because >> >>>>>>>> we >> >>>>>>>>> are using NFS (V4 version) in some environments, so wanted to >> learn >> >>>>>>>> tunings >> >>>>>>>>> when we use NFS. >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> Ram >> >>>>>>>>> >> >>>>>>>>> On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < >> >>>>>>>>> [email protected]> wrote: >> >>>>>>>>> >> >>>>>>>>>> Sorry, i thought it's NFS but it's actually SAN storage volume. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> Ram >> >>>>>>>>>> >> >>>>>>>>>> On Fri, Nov 9, 2018, 2:10 AM Gordon Sim <[email protected] >> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> On 08/11/18 16:56, rammohan ganapavarapu wrote: >> >>>>>>>>>>>> I was wrong about the NFS for qpid journal files, looks like >> >>> they >> >>>>>>>> are on >> >>>>>>>>>>>> NFS, so does NFS cause this issue? >> >>>>>>>>>>> Yes, I believe it does. What version of NFS are you using? >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>>>>>>>> To unsubscribe, e-mail: [email protected] >> >>>>>>>>>>> For additional commands, e-mail: [email protected] >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>> --------------------------------------------------------------------- >> >>>>>>>> To unsubscribe, e-mail: [email protected] >> >>>>>>>> For additional commands, e-mail: [email protected] >> >>>>>>>> >> >>>>>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: [email protected] >> >>>>> For additional commands, e-mail: [email protected] >> >>>>> >> >>>>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe, e-mail: [email protected] >> >>> For additional commands, e-mail: [email protected] >> >>> >> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
