Re: qpid-cpp-0.35 errors
Kim, Any tools to read or dump messages from journal files? Thanks, Ram On Fri, Dec 7, 2018 at 11:32 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Kim, > > We have only one main queue and one dead letter queue, and we have 10 > produces and 12 consumers and producers pump 1k messages/sec. Below is > qpid-stat -q output when it stopped taking any more messages. > > bash-4.1# qpid-stat -q > Queues > queue dur autoDel excl msg > msgIn msgOut bytes bytesIn bytesOut cons bind > > = > ax-q-axgroup-001-consumer-group-001 Y 0 > 114k 114k 0 5.88g5.88g 11 2 > ax-q-axgroup-001-consumer-group-001-dlY 0 > 0 0 0 00 0 2 > d72e183e-f0df-457c-89a7-81a2cff509c8:0.0 YY0 > 0 0 0 00 1 2 > > > Thanks, > Ram > > > > > On Fri, Dec 7, 2018 at 9:00 AM Kim van der Riet > wrote: > >> This looks like a bug to me, and that is why I am keen to see a >> reproducer if you can find one. How many queues are there? How many >> producers and consumers are there for each queue? How are the consumers >> working? Are they configured as listeners, or do they poll for new >> messages? How frequently? How long does it take under these conditions >> for the error to occur typically? If I can get some kind of idea what >> the runtime conditions are, it will give me some idea where to look. >> >> If you set the broker to use INFO+ logging (log-enable=info+), then you >> should see some detail about the starting and recovery of the store when >> the broker starts, which should include this info. The store settings in >> the config file are global, so when you set a particular buffer >> configuration, all queues will use this. It should be reported during >> startup when using INFO+ level logging. Watch your log size, however, as >> using this level will make the logs big. >> >> On 12/5/18 5:06 PM, rammohan ganapavarapu wrote: >> > Kim, >> > >> > We have set wcache-page-size=128 in qpidd.conf, restarted broker and let >> > client recreated the queues fresh, we still getting this error, how do >> we >> > find if queues created by client actually have this >> wcache-page-size=128? >> > >> > 2018-12-05 21:18:16 [Protocol] error Connection >> > qpid.:5672-:17769 closed by error: Queue : >> > MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() >> threw >> > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned >> > partly completed (state ENQ_PART). (This data_tok: id=456535 state=NONE) >> > >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) >> > >> > Thanks, >> > Ram >> > >> > >> > >> > On Tue, Dec 4, 2018 at 8:18 AM rammohan ganapavarapu < >> > rammohanga...@gmail.com> wrote: >> > >> >> Kim, >> >> >> >> Thank you, i will play with that setting, please let me know if any >> other >> >> tunings will help. >> >> >> >> Ram >> >> >> >> On Wed, Nov 28, 2018 at 8:04 AM Kim van der Riet >> >> wrote: >> >> >> >>> The answer to your first question depends on what is more important to >> >>> you - low latency or high throughput. Messages to be persisted will >> >>> accumulate in a buffer page until it is full or until a timer is >> >>> triggered, then it will be written to disk. It is not until this >> happens >> >>> that the message will be acknowledged by the broker. If low latency is >> >>> important, then having smaller but more numerous buffer pages will >> mean >> >>> the messages will not wait for very long before being written to disk >> >>> and acknowledged as received. However this occurs at the cost of some >> >>> efficiency, which can affect throughput. If you have large volumes of >> >>> messages and the throughput is more important, then using fewer but >> >>> larger buffer pages will help you. >> >>> >> >>> Be aware, however, that the product of the size and number of pages is >> >>> the total memory that will be consumed and held by the broker for >> >>> buffering *per queue*. If you have a very large number of queues, then >> >>> you must watch out that you don't over-size your write buffers or else >> >>> you will run out of memory. >> >>> >> >>> While I cannot give you specific answers, as these depend on your >> >>> performance priorities, I suggest some trial-and-error if you want to >> >>> adjust these values. >> >>> >> >>> The Transaction Prepared List (TPL) is a special global queue for >> >>> persisting transaction boundaries. As this info is usually small and >> >>> relatively infrequent, the tpl-* settings apply to this queue only and >> >>> the user has the option to use different values than the regular >> queues. >> >>> If you don't use transactions, then this can be ignored. It is not a
Re: qpid-cpp-0.35 errors
Kim, We have only one main queue and one dead letter queue, and we have 10 produces and 12 consumers and producers pump 1k messages/sec. Below is qpid-stat -q output when it stopped taking any more messages. bash-4.1# qpid-stat -q Queues queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind = ax-q-axgroup-001-consumer-group-001 Y 0 114k 114k 0 5.88g5.88g 11 2 ax-q-axgroup-001-consumer-group-001-dlY 0 0 0 0 00 0 2 d72e183e-f0df-457c-89a7-81a2cff509c8:0.0 YY0 0 0 0 00 1 2 Thanks, Ram On Fri, Dec 7, 2018 at 9:00 AM Kim van der Riet wrote: > This looks like a bug to me, and that is why I am keen to see a > reproducer if you can find one. How many queues are there? How many > producers and consumers are there for each queue? How are the consumers > working? Are they configured as listeners, or do they poll for new > messages? How frequently? How long does it take under these conditions > for the error to occur typically? If I can get some kind of idea what > the runtime conditions are, it will give me some idea where to look. > > If you set the broker to use INFO+ logging (log-enable=info+), then you > should see some detail about the starting and recovery of the store when > the broker starts, which should include this info. The store settings in > the config file are global, so when you set a particular buffer > configuration, all queues will use this. It should be reported during > startup when using INFO+ level logging. Watch your log size, however, as > using this level will make the logs big. > > On 12/5/18 5:06 PM, rammohan ganapavarapu wrote: > > Kim, > > > > We have set wcache-page-size=128 in qpidd.conf, restarted broker and let > > client recreated the queues fresh, we still getting this error, how do we > > find if queues created by client actually have this wcache-page-size=128? > > > > 2018-12-05 21:18:16 [Protocol] error Connection > > qpid.:5672-:17769 closed by error: Queue : > > MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw > > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned > > partly completed (state ENQ_PART). (This data_tok: id=456535 state=NONE) > > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) > > > > Thanks, > > Ram > > > > > > > > On Tue, Dec 4, 2018 at 8:18 AM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> Kim, > >> > >> Thank you, i will play with that setting, please let me know if any > other > >> tunings will help. > >> > >> Ram > >> > >> On Wed, Nov 28, 2018 at 8:04 AM Kim van der Riet > >> wrote: > >> > >>> The answer to your first question depends on what is more important to > >>> you - low latency or high throughput. Messages to be persisted will > >>> accumulate in a buffer page until it is full or until a timer is > >>> triggered, then it will be written to disk. It is not until this > happens > >>> that the message will be acknowledged by the broker. If low latency is > >>> important, then having smaller but more numerous buffer pages will mean > >>> the messages will not wait for very long before being written to disk > >>> and acknowledged as received. However this occurs at the cost of some > >>> efficiency, which can affect throughput. If you have large volumes of > >>> messages and the throughput is more important, then using fewer but > >>> larger buffer pages will help you. > >>> > >>> Be aware, however, that the product of the size and number of pages is > >>> the total memory that will be consumed and held by the broker for > >>> buffering *per queue*. If you have a very large number of queues, then > >>> you must watch out that you don't over-size your write buffers or else > >>> you will run out of memory. > >>> > >>> While I cannot give you specific answers, as these depend on your > >>> performance priorities, I suggest some trial-and-error if you want to > >>> adjust these values. > >>> > >>> The Transaction Prepared List (TPL) is a special global queue for > >>> persisting transaction boundaries. As this info is usually small and > >>> relatively infrequent, the tpl-* settings apply to this queue only and > >>> the user has the option to use different values than the regular > queues. > >>> If you don't use transactions, then this can be ignored. It is not a > >>> queue that can be written to directly, but the store creates its own > >>> data that is saved in this queue. Adjusting the tpl-* settings depends > >>> only on the frequency of transactions in the user's application or > >>> use-case. > >>> > >>> Hope that helps, > >>> > >>> Kim van der Riet
Re: qpid-cpp-0.35 errors
This looks like a bug to me, and that is why I am keen to see a reproducer if you can find one. How many queues are there? How many producers and consumers are there for each queue? How are the consumers working? Are they configured as listeners, or do they poll for new messages? How frequently? How long does it take under these conditions for the error to occur typically? If I can get some kind of idea what the runtime conditions are, it will give me some idea where to look. If you set the broker to use INFO+ logging (log-enable=info+), then you should see some detail about the starting and recovery of the store when the broker starts, which should include this info. The store settings in the config file are global, so when you set a particular buffer configuration, all queues will use this. It should be reported during startup when using INFO+ level logging. Watch your log size, however, as using this level will make the logs big. On 12/5/18 5:06 PM, rammohan ganapavarapu wrote: Kim, We have set wcache-page-size=128 in qpidd.conf, restarted broker and let client recreated the queues fresh, we still getting this error, how do we find if queues created by client actually have this wcache-page-size=128? 2018-12-05 21:18:16 [Protocol] error Connection qpid.:5672-:17769 closed by error: Queue : MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=456535 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) Thanks, Ram On Tue, Dec 4, 2018 at 8:18 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Kim, Thank you, i will play with that setting, please let me know if any other tunings will help. Ram On Wed, Nov 28, 2018 at 8:04 AM Kim van der Riet wrote: The answer to your first question depends on what is more important to you - low latency or high throughput. Messages to be persisted will accumulate in a buffer page until it is full or until a timer is triggered, then it will be written to disk. It is not until this happens that the message will be acknowledged by the broker. If low latency is important, then having smaller but more numerous buffer pages will mean the messages will not wait for very long before being written to disk and acknowledged as received. However this occurs at the cost of some efficiency, which can affect throughput. If you have large volumes of messages and the throughput is more important, then using fewer but larger buffer pages will help you. Be aware, however, that the product of the size and number of pages is the total memory that will be consumed and held by the broker for buffering *per queue*. If you have a very large number of queues, then you must watch out that you don't over-size your write buffers or else you will run out of memory. While I cannot give you specific answers, as these depend on your performance priorities, I suggest some trial-and-error if you want to adjust these values. The Transaction Prepared List (TPL) is a special global queue for persisting transaction boundaries. As this info is usually small and relatively infrequent, the tpl-* settings apply to this queue only and the user has the option to use different values than the regular queues. If you don't use transactions, then this can be ignored. It is not a queue that can be written to directly, but the store creates its own data that is saved in this queue. Adjusting the tpl-* settings depends only on the frequency of transactions in the user's application or use-case. Hope that helps, Kim van der Riet On 11/27/18 4:44 PM, rammohan ganapavarapu wrote: Kim, 1. My message size is around 80kb, so what would be suggested values for the blow properties? wcache-page-size wcache-num-pages tpl-wcache-num-pages tpl-wcache-page-size right now i have all defaults, so i am trying to see if i can tune these values for my messages size to avoid those AIO busy cases. I have try to define those properties/options in qpidd.conf file but when i run qpid-config queues its not showing those values on my queues created by client application, do i have to define those options when i create queue instead of keep them in qpidd.conf? 2. What is difference b/w tpl-wcache-page-size and wcache-page-size Thanks, Ram On Fri, Nov 16, 2018 at 9:26 AM Kim van der Riet wrote: There is little documentation on linearstore. Certainly, the Apache docs don't contain much. I think this is an oversight, but it won't get fixed anytime soon. Kim On 11/16/18 12:11 PM, rammohan ganapavarapu wrote: Any one point me to the doc where i can read internals about how linearstore works and how qpid uses it? Thanks, Ram On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Kim, Thanks for clearing that up for me, does it support SAN storage
Re: qpid-cpp-0.35 errors
Kim, We have set wcache-page-size=128 in qpidd.conf, restarted broker and let client recreated the queues fresh, we still getting this error, how do we find if queues created by client actually have this wcache-page-size=128? 2018-12-05 21:18:16 [Protocol] error Connection qpid.:5672-:17769 closed by error: Queue : MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=456535 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) Thanks, Ram On Tue, Dec 4, 2018 at 8:18 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Kim, > > Thank you, i will play with that setting, please let me know if any other > tunings will help. > > Ram > > On Wed, Nov 28, 2018 at 8:04 AM Kim van der Riet > wrote: > >> The answer to your first question depends on what is more important to >> you - low latency or high throughput. Messages to be persisted will >> accumulate in a buffer page until it is full or until a timer is >> triggered, then it will be written to disk. It is not until this happens >> that the message will be acknowledged by the broker. If low latency is >> important, then having smaller but more numerous buffer pages will mean >> the messages will not wait for very long before being written to disk >> and acknowledged as received. However this occurs at the cost of some >> efficiency, which can affect throughput. If you have large volumes of >> messages and the throughput is more important, then using fewer but >> larger buffer pages will help you. >> >> Be aware, however, that the product of the size and number of pages is >> the total memory that will be consumed and held by the broker for >> buffering *per queue*. If you have a very large number of queues, then >> you must watch out that you don't over-size your write buffers or else >> you will run out of memory. >> >> While I cannot give you specific answers, as these depend on your >> performance priorities, I suggest some trial-and-error if you want to >> adjust these values. >> >> The Transaction Prepared List (TPL) is a special global queue for >> persisting transaction boundaries. As this info is usually small and >> relatively infrequent, the tpl-* settings apply to this queue only and >> the user has the option to use different values than the regular queues. >> If you don't use transactions, then this can be ignored. It is not a >> queue that can be written to directly, but the store creates its own >> data that is saved in this queue. Adjusting the tpl-* settings depends >> only on the frequency of transactions in the user's application or >> use-case. >> >> Hope that helps, >> >> Kim van der Riet >> >> On 11/27/18 4:44 PM, rammohan ganapavarapu wrote: >> > Kim, >> > >> > 1. My message size is around 80kb, so what would be suggested values for >> > the blow properties? >> > >> > >> > wcache-page-size >> > wcache-num-pages >> > tpl-wcache-num-pages >> > tpl-wcache-page-size >> > >> > right now i have all defaults, so i am trying to see if i can tune these >> > values for my messages size to avoid those AIO busy cases. I have try >> to >> > define those properties/options in qpidd.conf file but when i run >> > qpid-config queues its not showing those values on my queues created by >> > client application, do i have to define those options when i create >> queue >> > instead of keep them in qpidd.conf? >> > >> > 2. What is difference b/w tpl-wcache-page-size and wcache-page-size >> > >> > Thanks, >> > Ram >> > >> > On Fri, Nov 16, 2018 at 9:26 AM Kim van der Riet >> > wrote: >> > >> >> There is little documentation on linearstore. Certainly, the Apache >> docs >> >> don't contain much. I think this is an oversight, but it won't get >> fixed >> >> anytime soon. >> >> >> >> Kim >> >> >> >> On 11/16/18 12:11 PM, rammohan ganapavarapu wrote: >> >>> Any one point me to the doc where i can read internals about how >> >>> linearstore works and how qpid uses it? >> >>> >> >>> Thanks, >> >>> Ram >> >>> >> >>> On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < >> >>> rammohanga...@gmail.com> wrote: >> >>> >> Kim, >> >> Thanks for clearing that up for me, does it support SAN storage >> blocks. >> Where can i read more about linearstore if i want to know the low >> level >> internals? >> >> Ram >> >> On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet < >> kvand...@redhat.com> >> wrote: >> >> > The linearstore relies on using libaio for its async disk writes. >> The >> > O_DIRECT flag is used, and this requires a block of aligned memory >> to >> > serve as a memory buffer for disk write operations. To my knowledge, >> > this technique only works with local disks and controllers. NFS does >> >> not >> > allow for DMA memory writes to disk AFAIK, and for as long as I
Re: qpid-cpp-0.35 errors
Kim, Thank you, i will play with that setting, please let me know if any other tunings will help. Ram On Wed, Nov 28, 2018 at 8:04 AM Kim van der Riet wrote: > The answer to your first question depends on what is more important to > you - low latency or high throughput. Messages to be persisted will > accumulate in a buffer page until it is full or until a timer is > triggered, then it will be written to disk. It is not until this happens > that the message will be acknowledged by the broker. If low latency is > important, then having smaller but more numerous buffer pages will mean > the messages will not wait for very long before being written to disk > and acknowledged as received. However this occurs at the cost of some > efficiency, which can affect throughput. If you have large volumes of > messages and the throughput is more important, then using fewer but > larger buffer pages will help you. > > Be aware, however, that the product of the size and number of pages is > the total memory that will be consumed and held by the broker for > buffering *per queue*. If you have a very large number of queues, then > you must watch out that you don't over-size your write buffers or else > you will run out of memory. > > While I cannot give you specific answers, as these depend on your > performance priorities, I suggest some trial-and-error if you want to > adjust these values. > > The Transaction Prepared List (TPL) is a special global queue for > persisting transaction boundaries. As this info is usually small and > relatively infrequent, the tpl-* settings apply to this queue only and > the user has the option to use different values than the regular queues. > If you don't use transactions, then this can be ignored. It is not a > queue that can be written to directly, but the store creates its own > data that is saved in this queue. Adjusting the tpl-* settings depends > only on the frequency of transactions in the user's application or > use-case. > > Hope that helps, > > Kim van der Riet > > On 11/27/18 4:44 PM, rammohan ganapavarapu wrote: > > Kim, > > > > 1. My message size is around 80kb, so what would be suggested values for > > the blow properties? > > > > > > wcache-page-size > > wcache-num-pages > > tpl-wcache-num-pages > > tpl-wcache-page-size > > > > right now i have all defaults, so i am trying to see if i can tune these > > values for my messages size to avoid those AIO busy cases. I have try to > > define those properties/options in qpidd.conf file but when i run > > qpid-config queues its not showing those values on my queues created by > > client application, do i have to define those options when i create queue > > instead of keep them in qpidd.conf? > > > > 2. What is difference b/w tpl-wcache-page-size and wcache-page-size > > > > Thanks, > > Ram > > > > On Fri, Nov 16, 2018 at 9:26 AM Kim van der Riet > > wrote: > > > >> There is little documentation on linearstore. Certainly, the Apache docs > >> don't contain much. I think this is an oversight, but it won't get fixed > >> anytime soon. > >> > >> Kim > >> > >> On 11/16/18 12:11 PM, rammohan ganapavarapu wrote: > >>> Any one point me to the doc where i can read internals about how > >>> linearstore works and how qpid uses it? > >>> > >>> Thanks, > >>> Ram > >>> > >>> On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < > >>> rammohanga...@gmail.com> wrote: > >>> > Kim, > > Thanks for clearing that up for me, does it support SAN storage > blocks. > Where can i read more about linearstore if i want to know the low > level > internals? > > Ram > > On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet > > wrote: > > > The linearstore relies on using libaio for its async disk writes. The > > O_DIRECT flag is used, and this requires a block of aligned memory to > > serve as a memory buffer for disk write operations. To my knowledge, > > this technique only works with local disks and controllers. NFS does > >> not > > allow for DMA memory writes to disk AFAIK, and for as long as I can > > remember, has been a problem for the linearstore. With some work it > > might be possible to make it work using another write technique > though. > > NFS has never been a "supported" medium for linearstore. > > > > On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: > >> But how does NFS will cause this issue, i am interested to see > because > > we > >> are using NFS (V4 version) in some environments, so wanted to learn > > tunings > >> when we use NFS. > >> > >> Thanks, > >> Ram > >> > >> On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < > >> rammohanga...@gmail.com> wrote: > >> > >>> Sorry, i thought it's NFS but it's actually SAN storage volume. > >>> > >>> Thanks, > >>> Ram > >>> > >>> On Fri, Nov 9, 2018, 2:10 AM Gordon Sim >>> > On 08/11/18 16:56, rammohan
Re: qpid-cpp-0.35 errors
The answer to your first question depends on what is more important to you - low latency or high throughput. Messages to be persisted will accumulate in a buffer page until it is full or until a timer is triggered, then it will be written to disk. It is not until this happens that the message will be acknowledged by the broker. If low latency is important, then having smaller but more numerous buffer pages will mean the messages will not wait for very long before being written to disk and acknowledged as received. However this occurs at the cost of some efficiency, which can affect throughput. If you have large volumes of messages and the throughput is more important, then using fewer but larger buffer pages will help you. Be aware, however, that the product of the size and number of pages is the total memory that will be consumed and held by the broker for buffering *per queue*. If you have a very large number of queues, then you must watch out that you don't over-size your write buffers or else you will run out of memory. While I cannot give you specific answers, as these depend on your performance priorities, I suggest some trial-and-error if you want to adjust these values. The Transaction Prepared List (TPL) is a special global queue for persisting transaction boundaries. As this info is usually small and relatively infrequent, the tpl-* settings apply to this queue only and the user has the option to use different values than the regular queues. If you don't use transactions, then this can be ignored. It is not a queue that can be written to directly, but the store creates its own data that is saved in this queue. Adjusting the tpl-* settings depends only on the frequency of transactions in the user's application or use-case. Hope that helps, Kim van der Riet On 11/27/18 4:44 PM, rammohan ganapavarapu wrote: Kim, 1. My message size is around 80kb, so what would be suggested values for the blow properties? wcache-page-size wcache-num-pages tpl-wcache-num-pages tpl-wcache-page-size right now i have all defaults, so i am trying to see if i can tune these values for my messages size to avoid those AIO busy cases. I have try to define those properties/options in qpidd.conf file but when i run qpid-config queues its not showing those values on my queues created by client application, do i have to define those options when i create queue instead of keep them in qpidd.conf? 2. What is difference b/w tpl-wcache-page-size and wcache-page-size Thanks, Ram On Fri, Nov 16, 2018 at 9:26 AM Kim van der Riet wrote: There is little documentation on linearstore. Certainly, the Apache docs don't contain much. I think this is an oversight, but it won't get fixed anytime soon. Kim On 11/16/18 12:11 PM, rammohan ganapavarapu wrote: Any one point me to the doc where i can read internals about how linearstore works and how qpid uses it? Thanks, Ram On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Kim, Thanks for clearing that up for me, does it support SAN storage blocks. Where can i read more about linearstore if i want to know the low level internals? Ram On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet wrote: The linearstore relies on using libaio for its async disk writes. The O_DIRECT flag is used, and this requires a block of aligned memory to serve as a memory buffer for disk write operations. To my knowledge, this technique only works with local disks and controllers. NFS does not allow for DMA memory writes to disk AFAIK, and for as long as I can remember, has been a problem for the linearstore. With some work it might be possible to make it work using another write technique though. NFS has never been a "supported" medium for linearstore. On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: But how does NFS will cause this issue, i am interested to see because we are using NFS (V4 version) in some environments, so wanted to learn tunings when we use NFS. Thanks, Ram On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Sorry, i thought it's NFS but it's actually SAN storage volume. Thanks, Ram On Fri, Nov 9, 2018, 2:10 AM Gordon Sim On 08/11/18 16:56, rammohan ganapavarapu wrote: I was wrong about the NFS for qpid journal files, looks like they are on NFS, so does NFS cause this issue? Yes, I believe it does. What version of NFS are you using? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional
Re: qpid-cpp-0.35 errors
Kim, 1. My message size is around 80kb, so what would be suggested values for the blow properties? wcache-page-size wcache-num-pages tpl-wcache-num-pages tpl-wcache-page-size right now i have all defaults, so i am trying to see if i can tune these values for my messages size to avoid those AIO busy cases. I have try to define those properties/options in qpidd.conf file but when i run qpid-config queues its not showing those values on my queues created by client application, do i have to define those options when i create queue instead of keep them in qpidd.conf? 2. What is difference b/w tpl-wcache-page-size and wcache-page-size Thanks, Ram On Fri, Nov 16, 2018 at 9:26 AM Kim van der Riet wrote: > There is little documentation on linearstore. Certainly, the Apache docs > don't contain much. I think this is an oversight, but it won't get fixed > anytime soon. > > Kim > > On 11/16/18 12:11 PM, rammohan ganapavarapu wrote: > > Any one point me to the doc where i can read internals about how > > linearstore works and how qpid uses it? > > > > Thanks, > > Ram > > > > On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> Kim, > >> > >> Thanks for clearing that up for me, does it support SAN storage blocks. > >> Where can i read more about linearstore if i want to know the low level > >> internals? > >> > >> Ram > >> > >> On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet > >> wrote: > >> > >>> The linearstore relies on using libaio for its async disk writes. The > >>> O_DIRECT flag is used, and this requires a block of aligned memory to > >>> serve as a memory buffer for disk write operations. To my knowledge, > >>> this technique only works with local disks and controllers. NFS does > not > >>> allow for DMA memory writes to disk AFAIK, and for as long as I can > >>> remember, has been a problem for the linearstore. With some work it > >>> might be possible to make it work using another write technique though. > >>> NFS has never been a "supported" medium for linearstore. > >>> > >>> On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: > But how does NFS will cause this issue, i am interested to see because > >>> we > are using NFS (V4 version) in some environments, so wanted to learn > >>> tunings > when we use NFS. > > Thanks, > Ram > > On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > > > Sorry, i thought it's NFS but it's actually SAN storage volume. > > > > Thanks, > > Ram > > > > On Fri, Nov 9, 2018, 2:10 AM Gordon Sim > > >> On 08/11/18 16:56, rammohan ganapavarapu wrote: > >>> I was wrong about the NFS for qpid journal files, looks like they > >>> are on > >>> NFS, so does NFS cause this issue? > >> Yes, I believe it does. What version of NFS are you using? > >> > >> > - > >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > >> For additional commands, e-mail: users-h...@qpid.apache.org > >> > >> > >>> - > >>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > >>> For additional commands, e-mail: users-h...@qpid.apache.org > >>> > >>> > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
There is little documentation on linearstore. Certainly, the Apache docs don't contain much. I think this is an oversight, but it won't get fixed anytime soon. Kim On 11/16/18 12:11 PM, rammohan ganapavarapu wrote: Any one point me to the doc where i can read internals about how linearstore works and how qpid uses it? Thanks, Ram On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Kim, Thanks for clearing that up for me, does it support SAN storage blocks. Where can i read more about linearstore if i want to know the low level internals? Ram On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet wrote: The linearstore relies on using libaio for its async disk writes. The O_DIRECT flag is used, and this requires a block of aligned memory to serve as a memory buffer for disk write operations. To my knowledge, this technique only works with local disks and controllers. NFS does not allow for DMA memory writes to disk AFAIK, and for as long as I can remember, has been a problem for the linearstore. With some work it might be possible to make it work using another write technique though. NFS has never been a "supported" medium for linearstore. On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: But how does NFS will cause this issue, i am interested to see because we are using NFS (V4 version) in some environments, so wanted to learn tunings when we use NFS. Thanks, Ram On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Sorry, i thought it's NFS but it's actually SAN storage volume. Thanks, Ram On Fri, Nov 9, 2018, 2:10 AM Gordon Sim On 08/11/18 16:56, rammohan ganapavarapu wrote: I was wrong about the NFS for qpid journal files, looks like they are on NFS, so does NFS cause this issue? Yes, I believe it does. What version of NFS are you using? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
Kim, Actually we are using qpid as part of our application, and customer is using my application. They are facing this issue and it is happening to them intermittently but still don't know at what scenario it is happening. I was trying the same application with NFS but still couldn't reproduce it. We took tcpdump and i see that tcp trace doesn't have full message and it is getting truncate before broker getting close the tcp connection. Thanks, Ram On Fri, Nov 16, 2018 at 8:33 AM Kim van der Riet wrote: > Did you find a reproducer at all? > > Kim > > On 11/12/18 11:43 AM, rammohan ganapavarapu wrote: > > Kim, > > > > Thanks for clearing that up for me, does it support SAN storage blocks. > > Where can i read more about linearstore if i want to know the low level > > internals? > > > > Ram > > > > On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet > > wrote: > > > >> The linearstore relies on using libaio for its async disk writes. The > >> O_DIRECT flag is used, and this requires a block of aligned memory to > >> serve as a memory buffer for disk write operations. To my knowledge, > >> this technique only works with local disks and controllers. NFS does not > >> allow for DMA memory writes to disk AFAIK, and for as long as I can > >> remember, has been a problem for the linearstore. With some work it > >> might be possible to make it work using another write technique though. > >> NFS has never been a "supported" medium for linearstore. > >> > >> On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: > >>> But how does NFS will cause this issue, i am interested to see because > we > >>> are using NFS (V4 version) in some environments, so wanted to learn > >> tunings > >>> when we use NFS. > >>> > >>> Thanks, > >>> Ram > >>> > >>> On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < > >>> rammohanga...@gmail.com> wrote: > >>> > Sorry, i thought it's NFS but it's actually SAN storage volume. > > Thanks, > Ram > > On Fri, Nov 9, 2018, 2:10 AM Gordon Sim > > On 08/11/18 16:56, rammohan ganapavarapu wrote: > >> I was wrong about the NFS for qpid journal files, looks like they > are > >> on > >> NFS, so does NFS cause this issue? > > Yes, I believe it does. What version of NFS are you using? > > > > - > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > > For additional commands, e-mail: users-h...@qpid.apache.org > > > > > >> - > >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > >> For additional commands, e-mail: users-h...@qpid.apache.org > >> > >> > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
Any one point me to the doc where i can read internals about how linearstore works and how qpid uses it? Thanks, Ram On Mon, Nov 12, 2018 at 8:43 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Kim, > > Thanks for clearing that up for me, does it support SAN storage blocks. > Where can i read more about linearstore if i want to know the low level > internals? > > Ram > > On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet > wrote: > >> The linearstore relies on using libaio for its async disk writes. The >> O_DIRECT flag is used, and this requires a block of aligned memory to >> serve as a memory buffer for disk write operations. To my knowledge, >> this technique only works with local disks and controllers. NFS does not >> allow for DMA memory writes to disk AFAIK, and for as long as I can >> remember, has been a problem for the linearstore. With some work it >> might be possible to make it work using another write technique though. >> NFS has never been a "supported" medium for linearstore. >> >> On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: >> > But how does NFS will cause this issue, i am interested to see because >> we >> > are using NFS (V4 version) in some environments, so wanted to learn >> tunings >> > when we use NFS. >> > >> > Thanks, >> > Ram >> > >> > On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < >> > rammohanga...@gmail.com> wrote: >> > >> >> Sorry, i thought it's NFS but it's actually SAN storage volume. >> >> >> >> Thanks, >> >> Ram >> >> >> >> On Fri, Nov 9, 2018, 2:10 AM Gordon Sim > >> >> >>> On 08/11/18 16:56, rammohan ganapavarapu wrote: >> I was wrong about the NFS for qpid journal files, looks like they >> are on >> NFS, so does NFS cause this issue? >> >>> Yes, I believe it does. What version of NFS are you using? >> >>> >> >>> - >> >>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> >>> For additional commands, e-mail: users-h...@qpid.apache.org >> >>> >> >>> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >>
Re: qpid-cpp-0.35 errors
Did you find a reproducer at all? Kim On 11/12/18 11:43 AM, rammohan ganapavarapu wrote: Kim, Thanks for clearing that up for me, does it support SAN storage blocks. Where can i read more about linearstore if i want to know the low level internals? Ram On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet wrote: The linearstore relies on using libaio for its async disk writes. The O_DIRECT flag is used, and this requires a block of aligned memory to serve as a memory buffer for disk write operations. To my knowledge, this technique only works with local disks and controllers. NFS does not allow for DMA memory writes to disk AFAIK, and for as long as I can remember, has been a problem for the linearstore. With some work it might be possible to make it work using another write technique though. NFS has never been a "supported" medium for linearstore. On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: But how does NFS will cause this issue, i am interested to see because we are using NFS (V4 version) in some environments, so wanted to learn tunings when we use NFS. Thanks, Ram On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Sorry, i thought it's NFS but it's actually SAN storage volume. Thanks, Ram On Fri, Nov 9, 2018, 2:10 AM Gordon Sim On 08/11/18 16:56, rammohan ganapavarapu wrote: I was wrong about the NFS for qpid journal files, looks like they are on NFS, so does NFS cause this issue? Yes, I believe it does. What version of NFS are you using? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
Kim, Thanks for clearing that up for me, does it support SAN storage blocks. Where can i read more about linearstore if i want to know the low level internals? Ram On Mon, Nov 12, 2018 at 8:32 AM Kim van der Riet wrote: > The linearstore relies on using libaio for its async disk writes. The > O_DIRECT flag is used, and this requires a block of aligned memory to > serve as a memory buffer for disk write operations. To my knowledge, > this technique only works with local disks and controllers. NFS does not > allow for DMA memory writes to disk AFAIK, and for as long as I can > remember, has been a problem for the linearstore. With some work it > might be possible to make it work using another write technique though. > NFS has never been a "supported" medium for linearstore. > > On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: > > But how does NFS will cause this issue, i am interested to see because we > > are using NFS (V4 version) in some environments, so wanted to learn > tunings > > when we use NFS. > > > > Thanks, > > Ram > > > > On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> Sorry, i thought it's NFS but it's actually SAN storage volume. > >> > >> Thanks, > >> Ram > >> > >> On Fri, Nov 9, 2018, 2:10 AM Gordon Sim >> > >>> On 08/11/18 16:56, rammohan ganapavarapu wrote: > I was wrong about the NFS for qpid journal files, looks like they are > on > NFS, so does NFS cause this issue? > >>> Yes, I believe it does. What version of NFS are you using? > >>> > >>> - > >>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > >>> For additional commands, e-mail: users-h...@qpid.apache.org > >>> > >>> > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
The linearstore relies on using libaio for its async disk writes. The O_DIRECT flag is used, and this requires a block of aligned memory to serve as a memory buffer for disk write operations. To my knowledge, this technique only works with local disks and controllers. NFS does not allow for DMA memory writes to disk AFAIK, and for as long as I can remember, has been a problem for the linearstore. With some work it might be possible to make it work using another write technique though. NFS has never been a "supported" medium for linearstore. On 11/9/18 4:28 PM, rammohan ganapavarapu wrote: But how does NFS will cause this issue, i am interested to see because we are using NFS (V4 version) in some environments, so wanted to learn tunings when we use NFS. Thanks, Ram On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Sorry, i thought it's NFS but it's actually SAN storage volume. Thanks, Ram On Fri, Nov 9, 2018, 2:10 AM Gordon Sim On 08/11/18 16:56, rammohan ganapavarapu wrote: I was wrong about the NFS for qpid journal files, looks like they are on NFS, so does NFS cause this issue? Yes, I believe it does. What version of NFS are you using? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
But how does NFS will cause this issue, i am interested to see because we are using NFS (V4 version) in some environments, so wanted to learn tunings when we use NFS. Thanks, Ram On Fri, Nov 9, 2018 at 6:48 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Sorry, i thought it's NFS but it's actually SAN storage volume. > > Thanks, > Ram > > On Fri, Nov 9, 2018, 2:10 AM Gordon Sim >> On 08/11/18 16:56, rammohan ganapavarapu wrote: >> > I was wrong about the NFS for qpid journal files, looks like they are on >> > NFS, so does NFS cause this issue? >> >> Yes, I believe it does. What version of NFS are you using? >> >> - >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >>
Re: qpid-cpp-0.35 errors
Sorry, i thought it's NFS but it's actually SAN storage volume. Thanks, Ram On Fri, Nov 9, 2018, 2:10 AM Gordon Sim On 08/11/18 16:56, rammohan ganapavarapu wrote: > > I was wrong about the NFS for qpid journal files, looks like they are on > > NFS, so does NFS cause this issue? > > Yes, I believe it does. What version of NFS are you using? > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
On 08/11/18 16:56, rammohan ganapavarapu wrote: I was wrong about the NFS for qpid journal files, looks like they are on NFS, so does NFS cause this issue? Yes, I believe it does. What version of NFS are you using? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
Kim/Gordon, I was wrong about the NFS for qpid journal files, looks like they are on NFS, so does NFS cause this issue? Ram On Wed, Nov 7, 2018 at 12:18 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Kim, > > Ok, i am still trying to see what part of my java application is causing > that issue, yes that issue is happening intermittently. Regarding > "JERR_WMGR_ENQDISCONT" error, may be they are chained exceptions from the > previous error JERR_JCNTL_AIOCMPLWAIT? > > Does message size contribute to this issue? > > Thanks, > Ram > > On Wed, Nov 7, 2018 at 11:37 AM Kim van der Riet > wrote: > >> No, they are not. >> >> These two defines govern the number of sleeps and the sleep time while >> waiting for before throwing an exception during recovery only. They do >> not play a role during normal operation. >> >> If you are able to compile the broker code, you can try playing with >> these values. But I don't think they will make much difference to the >> overall problem. I think some of the other errors you have been seeing >> prior to this one are closer to where the real problem lies - such as >> the JRNL_WMGR_ENQDISCONT error. >> >> Do you have a reproducer of any kind? Does this error occur predictably >> under some or other conditions? >> >> Thanks, >> >> Kim van der Riet >> >> On 11/7/18 12:51 PM, rammohan ganapavarapu wrote: >> > Kim, >> > >> > I see these two settings from code, can these be configurable? >> > >> > #define MAX_AIO_SLEEPS 10 // tot: ~1 sec >> > >> > #define AIO_SLEEP_TIME_US 10 // 0.01 ms >> > >> > >> > Ram >> > >> > On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu < >> > rammohanga...@gmail.com> wrote: >> > >> >> Thank you Kim, i will try your suggestions. >> >> >> >> On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet > wrote: >> >> >> >>> This error is a linearstore issue. It looks as though there is a >> single >> >>> write operation to disk that has become stuck, and is holding up all >> >>> further write operations. This happens because there is a fixed >> circular >> >>> pool of memory pages used for the AIO operations to disk, and when one >> >>> of these is "busy" (indicated by the A letter in the page state map), >> >>> write operations cannot continue until it is cleared. It it does not >> >>> clear within a certain time, then an exception is thrown, which >> usually >> >>> results in the broker closing the connection. >> >>> >> >>> The events leading up to a "stuck" write operation are complex and >> >>> sometimes difficult to reproduce. If you have a reproducer, then I >> would >> >>> be interested to see it! Even so, the ability to reproduce on another >> >>> machine is hard as it depends on such things as disk write speed, the >> >>> disk controller characteristics, the number of threads in the thread >> >>> pool (ie CPU type), memory and other hardware-related things. >> >>> >> >>> There are two linearstore parameters that you can try playing with to >> >>> see if you can change the behavior of the store: >> >>> >> >>> wcache-page-size: This sets the size of each page in the write buffer. >> >>> Larger page size is good for large messages, a smaller size will help >> if >> >>> you have small messages. >> >>> >> >>> wchache-num-pages: The total number of pages in the write buffer. >> >>> >> >>> Use the --help on the broker with the linearstore loaded to see more >> >>> details on this. I hope that helps a little. >> >>> >> >>> Kim van der Riet >> >>> >> >>> On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: >> Any help in understand why/when broker throws those errors and stop >> receiving message would be appreciated. >> >> Not sure if any kernel tuning or broker tuning needs to be done to >> solve this issue. >> >> Thanks in advance, >> Ram >> >> On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >> > Also from this log message (store level) it seems like waiting for >> AIO >> >>> to >> > complete. >> > >> > 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "> > name>": get_events() returned JERR_JCNTL_AIOCMPLWAIT; >> > wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF >> > ps=[-A--] >> > >> > page_state ps=[-A--] where A is >> >>> AIO_PENDING >> > aer=1 _aio_evt_rem; ///< Remaining AIO events >> > >> > When there is or there are pending AIO, does broker close the >> >>> connection? >> > is there any tuning that can be done to resolve this? >> > >> > Thanks, >> > Ram >> > >> > >> > >> > >> > On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < >> > rammohanga...@gmail.com> wrote: >> > >> >> I was check the code and i see these lines for that AIO timeout. >> >> >> >> case >> >>> qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: >> >>
Re: qpid-cpp-0.35 errors
On 11/7/18 3:18 PM, rammohan ganapavarapu wrote: Kim, Ok, i am still trying to see what part of my java application is causing that issue, yes that issue is happening intermittently. Regarding "JERR_WMGR_ENQDISCONT" error, may be they are chained exceptions from the previous error JERR_JCNTL_AIOCMPLWAIT? In my mind, it is more likely the other way around. But the logs should tell you that. It would be best to start with a clean store before each test so you don't inherit issues from a previous test or run. Does message size contribute to this issue? Yes, but only in the sense that the size alters the packing of the write buffers, and the timing of when they are written. Also, the number of simultaneous producers and consumers will affect this. In particular, when two consumers are simultaneously sending messages to the same queue, also if a consumer is consuming from a queue while a producer is also sending are going to be the main factors in any race condition such as I suspect this is. Playing with those will give clues as to what is happening. You could try the following, each time starting with a clean store: 1. Only allowing a single producer, followed by a single consumer (ie not at the same time); 2. Allowing a single producer and a single consumer to operate simultaneously; 3. Allowing multiple producers (I don't know if your use-case has this) only 4. Allowing multiple consumers. Once you have isolated which scenarios cause the problem, then try varying the message size. The answers to these will help isolating where the issue is happening. Thanks, Ram On Wed, Nov 7, 2018 at 11:37 AM Kim van der Riet wrote: No, they are not. These two defines govern the number of sleeps and the sleep time while waiting for before throwing an exception during recovery only. They do not play a role during normal operation. If you are able to compile the broker code, you can try playing with these values. But I don't think they will make much difference to the overall problem. I think some of the other errors you have been seeing prior to this one are closer to where the real problem lies - such as the JRNL_WMGR_ENQDISCONT error. Do you have a reproducer of any kind? Does this error occur predictably under some or other conditions? Thanks, Kim van der Riet On 11/7/18 12:51 PM, rammohan ganapavarapu wrote: Kim, I see these two settings from code, can these be configurable? #define MAX_AIO_SLEEPS 10 // tot: ~1 sec #define AIO_SLEEP_TIME_US 10 // 0.01 ms Ram On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Thank you Kim, i will try your suggestions. On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet wrote: This error is a linearstore issue. It looks as though there is a single write operation to disk that has become stuck, and is holding up all further write operations. This happens because there is a fixed circular pool of memory pages used for the AIO operations to disk, and when one of these is "busy" (indicated by the A letter in the page state map), write operations cannot continue until it is cleared. It it does not clear within a certain time, then an exception is thrown, which usually results in the broker closing the connection. The events leading up to a "stuck" write operation are complex and sometimes difficult to reproduce. If you have a reproducer, then I would be interested to see it! Even so, the ability to reproduce on another machine is hard as it depends on such things as disk write speed, the disk controller characteristics, the number of threads in the thread pool (ie CPU type), memory and other hardware-related things. There are two linearstore parameters that you can try playing with to see if you can change the behavior of the store: wcache-page-size: This sets the size of each page in the write buffer. Larger page size is good for large messages, a smaller size will help if you have small messages. wchache-num-pages: The total number of pages in the write buffer. Use the --help on the broker with the linearstore loaded to see more details on this. I hope that helps a little. Kim van der Riet On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: Any help in understand why/when broker throws those errors and stop receiving message would be appreciated. Not sure if any kernel tuning or broker tuning needs to be done to solve this issue. Thanks in advance, Ram On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Also from this log message (store level) it seems like waiting for AIO to complete. 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF ps=[-A--] page_state ps=[-A--] where A is AIO_PENDING aer=1 _aio_evt_rem; ///< Remaining AIO events When there is or there are pending
Re: qpid-cpp-0.35 errors
Do you have any kernel (net/disk) tuning recommendations for qpid-cpp with linearstore? Ram On Thu, Nov 8, 2018 at 8:56 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Kim/Gordon, > > I was wrong about the NFS for qpid journal files, looks like they are on > NFS, so does NFS cause this issue? > > Ram > > On Wed, Nov 7, 2018 at 12:18 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Kim, >> >> Ok, i am still trying to see what part of my java application is causing >> that issue, yes that issue is happening intermittently. Regarding >> "JERR_WMGR_ENQDISCONT" error, may be they are chained exceptions from the >> previous error JERR_JCNTL_AIOCMPLWAIT? >> >> Does message size contribute to this issue? >> >> Thanks, >> Ram >> >> On Wed, Nov 7, 2018 at 11:37 AM Kim van der Riet >> wrote: >> >>> No, they are not. >>> >>> These two defines govern the number of sleeps and the sleep time while >>> waiting for before throwing an exception during recovery only. They do >>> not play a role during normal operation. >>> >>> If you are able to compile the broker code, you can try playing with >>> these values. But I don't think they will make much difference to the >>> overall problem. I think some of the other errors you have been seeing >>> prior to this one are closer to where the real problem lies - such as >>> the JRNL_WMGR_ENQDISCONT error. >>> >>> Do you have a reproducer of any kind? Does this error occur predictably >>> under some or other conditions? >>> >>> Thanks, >>> >>> Kim van der Riet >>> >>> On 11/7/18 12:51 PM, rammohan ganapavarapu wrote: >>> > Kim, >>> > >>> > I see these two settings from code, can these be configurable? >>> > >>> > #define MAX_AIO_SLEEPS 10 // tot: ~1 sec >>> > >>> > #define AIO_SLEEP_TIME_US 10 // 0.01 ms >>> > >>> > >>> > Ram >>> > >>> > On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu < >>> > rammohanga...@gmail.com> wrote: >>> > >>> >> Thank you Kim, i will try your suggestions. >>> >> >>> >> On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet >> wrote: >>> >> >>> >>> This error is a linearstore issue. It looks as though there is a >>> single >>> >>> write operation to disk that has become stuck, and is holding up all >>> >>> further write operations. This happens because there is a fixed >>> circular >>> >>> pool of memory pages used for the AIO operations to disk, and when >>> one >>> >>> of these is "busy" (indicated by the A letter in the page state >>> map), >>> >>> write operations cannot continue until it is cleared. It it does not >>> >>> clear within a certain time, then an exception is thrown, which >>> usually >>> >>> results in the broker closing the connection. >>> >>> >>> >>> The events leading up to a "stuck" write operation are complex and >>> >>> sometimes difficult to reproduce. If you have a reproducer, then I >>> would >>> >>> be interested to see it! Even so, the ability to reproduce on another >>> >>> machine is hard as it depends on such things as disk write speed, the >>> >>> disk controller characteristics, the number of threads in the thread >>> >>> pool (ie CPU type), memory and other hardware-related things. >>> >>> >>> >>> There are two linearstore parameters that you can try playing with to >>> >>> see if you can change the behavior of the store: >>> >>> >>> >>> wcache-page-size: This sets the size of each page in the write >>> buffer. >>> >>> Larger page size is good for large messages, a smaller size will >>> help if >>> >>> you have small messages. >>> >>> >>> >>> wchache-num-pages: The total number of pages in the write buffer. >>> >>> >>> >>> Use the --help on the broker with the linearstore loaded to see more >>> >>> details on this. I hope that helps a little. >>> >>> >>> >>> Kim van der Riet >>> >>> >>> >>> On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: >>> Any help in understand why/when broker throws those errors and stop >>> receiving message would be appreciated. >>> >>> Not sure if any kernel tuning or broker tuning needs to be done to >>> solve this issue. >>> >>> Thanks in advance, >>> Ram >>> >>> On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < >>> rammohanga...@gmail.com> wrote: >>> >>> > Also from this log message (store level) it seems like waiting for >>> AIO >>> >>> to >>> > complete. >>> > >>> > 2018-10-28 12:27:01 [Store] critical Linear Store: Journal >>> ">> > name>": get_events() returned JERR_JCNTL_AIOCMPLWAIT; >>> > wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF >>> > ps=[-A--] >>> > >>> > page_state ps=[-A--] where A is >>> >>> AIO_PENDING >>> > aer=1 _aio_evt_rem; ///< Remaining AIO events >>> > >>> > When there is or there are pending AIO, does broker close the >>> >>> connection? >>> > is there any tuning that can be done to resolve this? >>> > >>> > Thanks, >>> > Ram >>> > >>>
Re: qpid-cpp-0.35 errors
Resending, did not show up on the list the first time I sent it... Forwarded Message Subject:Re: qpid-cpp-0.35 errors Date: Thu, 8 Nov 2018 09:30:24 -0500 From: Kim van der Riet To: users@qpid.apache.org On 11/7/18 3:18 PM, rammohan ganapavarapu wrote: Kim, Ok, i am still trying to see what part of my java application is causing that issue, yes that issue is happening intermittently. Regarding "JERR_WMGR_ENQDISCONT" error, may be they are chained exceptions from the previous error JERR_JCNTL_AIOCMPLWAIT? In my mind, it is more likely the other way around. But the logs should tell you that. It would be best to start with a clean store before each test so you don't inherit issues from a previous test or run. Does message size contribute to this issue? Yes, but only in the sense that the size alters the packing of the write buffers, and the timing of when they are written. Also, the number of simultaneous producers and consumers will affect this. In particular, when two consumers are simultaneously sending messages to the same queue, also if a consumer is consuming from a queue while a producer is also sending are going to be the main factors in any race condition such as I suspect this is. Playing with those will give clues as to what is happening. You could try the following, each time starting with a clean store: 1. Only allowing a single producer, followed by a single consumer (ie not at the same time); 2. Allowing a single producer and a single consumer to operate simultaneously; 3. Allowing multiple producers (I don't know if your use-case has this) only 4. Allowing multiple consumers. Once you have isolated which scenarios cause the problem, then try varying the message size. The answers to these will help isolating where the issue is happening. Thanks, Ram On Wed, Nov 7, 2018 at 11:37 AM Kim van der Riet wrote: No, they are not. These two defines govern the number of sleeps and the sleep time while waiting for before throwing an exception during recovery only. They do not play a role during normal operation. If you are able to compile the broker code, you can try playing with these values. But I don't think they will make much difference to the overall problem. I think some of the other errors you have been seeing prior to this one are closer to where the real problem lies - such as the JRNL_WMGR_ENQDISCONT error. Do you have a reproducer of any kind? Does this error occur predictably under some or other conditions? Thanks, Kim van der Riet On 11/7/18 12:51 PM, rammohan ganapavarapu wrote: Kim, I see these two settings from code, can these be configurable? #define MAX_AIO_SLEEPS 10 // tot: ~1 sec #define AIO_SLEEP_TIME_US 10 // 0.01 ms Ram On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Thank you Kim, i will try your suggestions. On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet wrote: This error is a linearstore issue. It looks as though there is a single write operation to disk that has become stuck, and is holding up all further write operations. This happens because there is a fixed circular pool of memory pages used for the AIO operations to disk, and when one of these is "busy" (indicated by the A letter in the page state map), write operations cannot continue until it is cleared. It it does not clear within a certain time, then an exception is thrown, which usually results in the broker closing the connection. The events leading up to a "stuck" write operation are complex and sometimes difficult to reproduce. If you have a reproducer, then I would be interested to see it! Even so, the ability to reproduce on another machine is hard as it depends on such things as disk write speed, the disk controller characteristics, the number of threads in the thread pool (ie CPU type), memory and other hardware-related things. There are two linearstore parameters that you can try playing with to see if you can change the behavior of the store: wcache-page-size: This sets the size of each page in the write buffer. Larger page size is good for large messages, a smaller size will help if you have small messages. wchache-num-pages: The total number of pages in the write buffer. Use the --help on the broker with the linearstore loaded to see more details on this. I hope that helps a little. Kim van der Riet On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: Any help in understand why/when broker throws those errors and stop receiving message would be appreciated. Not sure if any kernel tuning or broker tuning needs to be done to solve this issue. Thanks in advance, Ram On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Also from this log message (store level) it seems like waiting for AIO to complete. 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_e
Re: qpid-cpp-0.35 errors
Kim, Ok, i am still trying to see what part of my java application is causing that issue, yes that issue is happening intermittently. Regarding "JERR_WMGR_ENQDISCONT" error, may be they are chained exceptions from the previous error JERR_JCNTL_AIOCMPLWAIT? Does message size contribute to this issue? Thanks, Ram On Wed, Nov 7, 2018 at 11:37 AM Kim van der Riet wrote: > No, they are not. > > These two defines govern the number of sleeps and the sleep time while > waiting for before throwing an exception during recovery only. They do > not play a role during normal operation. > > If you are able to compile the broker code, you can try playing with > these values. But I don't think they will make much difference to the > overall problem. I think some of the other errors you have been seeing > prior to this one are closer to where the real problem lies - such as > the JRNL_WMGR_ENQDISCONT error. > > Do you have a reproducer of any kind? Does this error occur predictably > under some or other conditions? > > Thanks, > > Kim van der Riet > > On 11/7/18 12:51 PM, rammohan ganapavarapu wrote: > > Kim, > > > > I see these two settings from code, can these be configurable? > > > > #define MAX_AIO_SLEEPS 10 // tot: ~1 sec > > > > #define AIO_SLEEP_TIME_US 10 // 0.01 ms > > > > > > Ram > > > > On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> Thank you Kim, i will try your suggestions. > >> > >> On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet wrote: > >> > >>> This error is a linearstore issue. It looks as though there is a single > >>> write operation to disk that has become stuck, and is holding up all > >>> further write operations. This happens because there is a fixed > circular > >>> pool of memory pages used for the AIO operations to disk, and when one > >>> of these is "busy" (indicated by the A letter in the page state map), > >>> write operations cannot continue until it is cleared. It it does not > >>> clear within a certain time, then an exception is thrown, which usually > >>> results in the broker closing the connection. > >>> > >>> The events leading up to a "stuck" write operation are complex and > >>> sometimes difficult to reproduce. If you have a reproducer, then I > would > >>> be interested to see it! Even so, the ability to reproduce on another > >>> machine is hard as it depends on such things as disk write speed, the > >>> disk controller characteristics, the number of threads in the thread > >>> pool (ie CPU type), memory and other hardware-related things. > >>> > >>> There are two linearstore parameters that you can try playing with to > >>> see if you can change the behavior of the store: > >>> > >>> wcache-page-size: This sets the size of each page in the write buffer. > >>> Larger page size is good for large messages, a smaller size will help > if > >>> you have small messages. > >>> > >>> wchache-num-pages: The total number of pages in the write buffer. > >>> > >>> Use the --help on the broker with the linearstore loaded to see more > >>> details on this. I hope that helps a little. > >>> > >>> Kim van der Riet > >>> > >>> On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: > Any help in understand why/when broker throws those errors and stop > receiving message would be appreciated. > > Not sure if any kernel tuning or broker tuning needs to be done to > solve this issue. > > Thanks in advance, > Ram > > On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > > > Also from this log message (store level) it seems like waiting for > AIO > >>> to > > complete. > > > > 2018-10-28 12:27:01 [Store] critical Linear Store: Journal " > name>": get_events() returned JERR_JCNTL_AIOCMPLWAIT; > > wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF > > ps=[-A--] > > > > page_state ps=[-A--] where A is > >>> AIO_PENDING > > aer=1 _aio_evt_rem; ///< Remaining AIO events > > > > When there is or there are pending AIO, does broker close the > >>> connection? > > is there any tuning that can be done to resolve this? > > > > Thanks, > > Ram > > > > > > > > > > On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> I was check the code and i see these lines for that AIO timeout. > >> > >> case > >>> qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: > >> if (++aio_sleep_cnt > MAX_AIO_SLEEPS) > >> THROW_STORE_EXCEPTION("Timeout waiting for > AIO in > >> MessageStoreImpl::recoverMessages()"); > >> ::usleep(AIO_SLEEP_TIME_US); > >> break; > >> > >> And these are the defaults > >> > >> #define MAX_AIO_SLEEPS 10 // tot: ~1
Re: qpid-cpp-0.35 errors
No, they are not. These two defines govern the number of sleeps and the sleep time while waiting for before throwing an exception during recovery only. They do not play a role during normal operation. If you are able to compile the broker code, you can try playing with these values. But I don't think they will make much difference to the overall problem. I think some of the other errors you have been seeing prior to this one are closer to where the real problem lies - such as the JRNL_WMGR_ENQDISCONT error. Do you have a reproducer of any kind? Does this error occur predictably under some or other conditions? Thanks, Kim van der Riet On 11/7/18 12:51 PM, rammohan ganapavarapu wrote: Kim, I see these two settings from code, can these be configurable? #define MAX_AIO_SLEEPS 10 // tot: ~1 sec #define AIO_SLEEP_TIME_US 10 // 0.01 ms Ram On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Thank you Kim, i will try your suggestions. On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet This error is a linearstore issue. It looks as though there is a single write operation to disk that has become stuck, and is holding up all further write operations. This happens because there is a fixed circular pool of memory pages used for the AIO operations to disk, and when one of these is "busy" (indicated by the A letter in the page state map), write operations cannot continue until it is cleared. It it does not clear within a certain time, then an exception is thrown, which usually results in the broker closing the connection. The events leading up to a "stuck" write operation are complex and sometimes difficult to reproduce. If you have a reproducer, then I would be interested to see it! Even so, the ability to reproduce on another machine is hard as it depends on such things as disk write speed, the disk controller characteristics, the number of threads in the thread pool (ie CPU type), memory and other hardware-related things. There are two linearstore parameters that you can try playing with to see if you can change the behavior of the store: wcache-page-size: This sets the size of each page in the write buffer. Larger page size is good for large messages, a smaller size will help if you have small messages. wchache-num-pages: The total number of pages in the write buffer. Use the --help on the broker with the linearstore loaded to see more details on this. I hope that helps a little. Kim van der Riet On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: Any help in understand why/when broker throws those errors and stop receiving message would be appreciated. Not sure if any kernel tuning or broker tuning needs to be done to solve this issue. Thanks in advance, Ram On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Also from this log message (store level) it seems like waiting for AIO to complete. 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF ps=[-A--] page_state ps=[-A--] where A is AIO_PENDING aer=1 _aio_evt_rem; ///< Remaining AIO events When there is or there are pending AIO, does broker close the connection? is there any tuning that can be done to resolve this? Thanks, Ram On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: I was check the code and i see these lines for that AIO timeout. case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: if (++aio_sleep_cnt > MAX_AIO_SLEEPS) THROW_STORE_EXCEPTION("Timeout waiting for AIO in MessageStoreImpl::recoverMessages()"); ::usleep(AIO_SLEEP_TIME_US); break; And these are the defaults #define MAX_AIO_SLEEPS 10 // tot: ~1 sec #define AIO_SLEEP_TIME_US 10 // 0.01 ms RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is waiting for AIO. So does page got blocked and its waiting for page availability? Ram On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we see this message 2018-10-27 18:58:25 [Store] warning Linear Store: Journal "": Bad record alignment found at fid=0x4605b offs=0x107680 (likely journal overwrite boundary); 19 filler record(s) required. 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": Recover phase write: Wrote filler record: fid=0x4605b offs=0x107680 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": Recover phase write: Wr... few more Recover phase logs It worked fine for a day and started throwing this message: 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 po=0
Re: qpid-cpp-0.35 errors
Kim, I see these two settings from code, can these be configurable? #define MAX_AIO_SLEEPS 10 // tot: ~1 sec #define AIO_SLEEP_TIME_US 10 // 0.01 ms Ram On Wed, Nov 7, 2018 at 7:04 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Thank you Kim, i will try your suggestions. > > On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet >> This error is a linearstore issue. It looks as though there is a single >> write operation to disk that has become stuck, and is holding up all >> further write operations. This happens because there is a fixed circular >> pool of memory pages used for the AIO operations to disk, and when one >> of these is "busy" (indicated by the A letter in the page state map), >> write operations cannot continue until it is cleared. It it does not >> clear within a certain time, then an exception is thrown, which usually >> results in the broker closing the connection. >> >> The events leading up to a "stuck" write operation are complex and >> sometimes difficult to reproduce. If you have a reproducer, then I would >> be interested to see it! Even so, the ability to reproduce on another >> machine is hard as it depends on such things as disk write speed, the >> disk controller characteristics, the number of threads in the thread >> pool (ie CPU type), memory and other hardware-related things. >> >> There are two linearstore parameters that you can try playing with to >> see if you can change the behavior of the store: >> >> wcache-page-size: This sets the size of each page in the write buffer. >> Larger page size is good for large messages, a smaller size will help if >> you have small messages. >> >> wchache-num-pages: The total number of pages in the write buffer. >> >> Use the --help on the broker with the linearstore loaded to see more >> details on this. I hope that helps a little. >> >> Kim van der Riet >> >> On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: >> > Any help in understand why/when broker throws those errors and stop >> > receiving message would be appreciated. >> > >> > Not sure if any kernel tuning or broker tuning needs to be done to >> > solve this issue. >> > >> > Thanks in advance, >> > Ram >> > >> > On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < >> > rammohanga...@gmail.com> wrote: >> > >> >> Also from this log message (store level) it seems like waiting for AIO >> to >> >> complete. >> >> >> >> 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "> >> name>": get_events() returned JERR_JCNTL_AIOCMPLWAIT; >> >> wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF >> >> ps=[-A--] >> >> >> >> page_state ps=[-A--] where A is >> AIO_PENDING >> >> aer=1 _aio_evt_rem; ///< Remaining AIO events >> >> >> >> When there is or there are pending AIO, does broker close the >> connection? >> >> is there any tuning that can be done to resolve this? >> >> >> >> Thanks, >> >> Ram >> >> >> >> >> >> >> >> >> >> On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < >> >> rammohanga...@gmail.com> wrote: >> >> >> >>> I was check the code and i see these lines for that AIO timeout. >> >>> >> >>>case >> qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: >> >>> if (++aio_sleep_cnt > MAX_AIO_SLEEPS) >> >>> THROW_STORE_EXCEPTION("Timeout waiting for AIO in >> >>> MessageStoreImpl::recoverMessages()"); >> >>> ::usleep(AIO_SLEEP_TIME_US); >> >>> break; >> >>> >> >>> And these are the defaults >> >>> >> >>> #define MAX_AIO_SLEEPS 10 // tot: ~1 sec >> >>> >> >>> #define AIO_SLEEP_TIME_US 10 // 0.01 ms >> >>> >> >>> >> >>>RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is >> >>> waiting for AIO. >> >>> >> >>> >> >>> >> >>> So does page got blocked and its waiting for page availability? >> >>> >> >>> >> >>> Ram >> >>> >> >>> On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < >> >>> rammohanga...@gmail.com> wrote: >> >>> >> Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that >> we >> see this message >> >> 2018-10-27 18:58:25 [Store] warning Linear Store: Journal >> "": Bad record alignment found at fid=0x4605b >> offs=0x107680 >> (likely journal overwrite boundary); 19 filler record(s) required. >> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >> "": Recover phase write: Wrote filler record: >> fid=0x4605b >> offs=0x107680 >> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >> "": Recover phase write: Wr... few more Recover phase >> logs >> >> It worked fine for a day and started throwing this message: >> >> 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": >> get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: >> pi=25 pc=8 >> po=0 aer=1 edac=TFFF ps=[-A--] >> 2018-10-28 12:27:01 [Broker] warning Exchange
Re: qpid-cpp-0.35 errors
Thank you Kim, i will try your suggestions. On Wed, Nov 7, 2018, 6:58 AM Kim van der Riet This error is a linearstore issue. It looks as though there is a single > write operation to disk that has become stuck, and is holding up all > further write operations. This happens because there is a fixed circular > pool of memory pages used for the AIO operations to disk, and when one > of these is "busy" (indicated by the A letter in the page state map), > write operations cannot continue until it is cleared. It it does not > clear within a certain time, then an exception is thrown, which usually > results in the broker closing the connection. > > The events leading up to a "stuck" write operation are complex and > sometimes difficult to reproduce. If you have a reproducer, then I would > be interested to see it! Even so, the ability to reproduce on another > machine is hard as it depends on such things as disk write speed, the > disk controller characteristics, the number of threads in the thread > pool (ie CPU type), memory and other hardware-related things. > > There are two linearstore parameters that you can try playing with to > see if you can change the behavior of the store: > > wcache-page-size: This sets the size of each page in the write buffer. > Larger page size is good for large messages, a smaller size will help if > you have small messages. > > wchache-num-pages: The total number of pages in the write buffer. > > Use the --help on the broker with the linearstore loaded to see more > details on this. I hope that helps a little. > > Kim van der Riet > > On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: > > Any help in understand why/when broker throws those errors and stop > > receiving message would be appreciated. > > > > Not sure if any kernel tuning or broker tuning needs to be done to > > solve this issue. > > > > Thanks in advance, > > Ram > > > > On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> Also from this log message (store level) it seems like waiting for AIO > to > >> complete. > >> > >> 2018-10-28 12:27:01 [Store] critical Linear Store: Journal " >> name>": get_events() returned JERR_JCNTL_AIOCMPLWAIT; > >> wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF > >> ps=[-A--] > >> > >> page_state ps=[-A--] where A is AIO_PENDING > >> aer=1 _aio_evt_rem; ///< Remaining AIO events > >> > >> When there is or there are pending AIO, does broker close the > connection? > >> is there any tuning that can be done to resolve this? > >> > >> Thanks, > >> Ram > >> > >> > >> > >> > >> On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < > >> rammohanga...@gmail.com> wrote: > >> > >>> I was check the code and i see these lines for that AIO timeout. > >>> > >>>case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: > >>> if (++aio_sleep_cnt > MAX_AIO_SLEEPS) > >>> THROW_STORE_EXCEPTION("Timeout waiting for AIO in > >>> MessageStoreImpl::recoverMessages()"); > >>> ::usleep(AIO_SLEEP_TIME_US); > >>> break; > >>> > >>> And these are the defaults > >>> > >>> #define MAX_AIO_SLEEPS 10 // tot: ~1 sec > >>> > >>> #define AIO_SLEEP_TIME_US 10 // 0.01 ms > >>> > >>> > >>>RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is > >>> waiting for AIO. > >>> > >>> > >>> > >>> So does page got blocked and its waiting for page availability? > >>> > >>> > >>> Ram > >>> > >>> On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < > >>> rammohanga...@gmail.com> wrote: > >>> > Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we > see this message > > 2018-10-27 18:58:25 [Store] warning Linear Store: Journal > "": Bad record alignment found at fid=0x4605b > offs=0x107680 > (likely journal overwrite boundary); 19 filler record(s) required. > 2018-10-27 18:58:25 [Store] notice Linear Store: Journal > "": Recover phase write: Wrote filler record: > fid=0x4605b > offs=0x107680 > 2018-10-27 18:58:25 [Store] notice Linear Store: Journal > "": Recover phase write: Wr... few more Recover phase > logs > > It worked fine for a day and started throwing this message: > > 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": > get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: > pi=25 pc=8 > po=0 aer=1 edac=TFFF ps=[-A--] > 2018-10-28 12:27:01 [Broker] warning Exchange cannot deliver to > queue : Queue : MessageStoreImpl::store() > failed: > jexception 0x0202 jcntl::handle_aio_wait() threw > JERR_JCNTL_AIOCMPLWAIT: > Timeout waiting for AIOs to complete. > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > 2018-10-28 12:27:01 [Broker] error Connection exception: >
Re: qpid-cpp-0.35 errors
This error is a linearstore issue. It looks as though there is a single write operation to disk that has become stuck, and is holding up all further write operations. This happens because there is a fixed circular pool of memory pages used for the AIO operations to disk, and when one of these is "busy" (indicated by the A letter in the page state map), write operations cannot continue until it is cleared. It it does not clear within a certain time, then an exception is thrown, which usually results in the broker closing the connection. The events leading up to a "stuck" write operation are complex and sometimes difficult to reproduce. If you have a reproducer, then I would be interested to see it! Even so, the ability to reproduce on another machine is hard as it depends on such things as disk write speed, the disk controller characteristics, the number of threads in the thread pool (ie CPU type), memory and other hardware-related things. There are two linearstore parameters that you can try playing with to see if you can change the behavior of the store: wcache-page-size: This sets the size of each page in the write buffer. Larger page size is good for large messages, a smaller size will help if you have small messages. wchache-num-pages: The total number of pages in the write buffer. Use the --help on the broker with the linearstore loaded to see more details on this. I hope that helps a little. Kim van der Riet On 11/6/18 2:12 PM, rammohan ganapavarapu wrote: Any help in understand why/when broker throws those errors and stop receiving message would be appreciated. Not sure if any kernel tuning or broker tuning needs to be done to solve this issue. Thanks in advance, Ram On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Also from this log message (store level) it seems like waiting for AIO to complete. 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF ps=[-A--] page_state ps=[-A--] where A is AIO_PENDING aer=1 _aio_evt_rem; ///< Remaining AIO events When there is or there are pending AIO, does broker close the connection? is there any tuning that can be done to resolve this? Thanks, Ram On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: I was check the code and i see these lines for that AIO timeout. case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: if (++aio_sleep_cnt > MAX_AIO_SLEEPS) THROW_STORE_EXCEPTION("Timeout waiting for AIO in MessageStoreImpl::recoverMessages()"); ::usleep(AIO_SLEEP_TIME_US); break; And these are the defaults #define MAX_AIO_SLEEPS 10 // tot: ~1 sec #define AIO_SLEEP_TIME_US 10 // 0.01 ms RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is waiting for AIO. So does page got blocked and its waiting for page availability? Ram On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we see this message 2018-10-27 18:58:25 [Store] warning Linear Store: Journal "": Bad record alignment found at fid=0x4605b offs=0x107680 (likely journal overwrite boundary); 19 filler record(s) required. 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": Recover phase write: Wrote filler record: fid=0x4605b offs=0x107680 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": Recover phase write: Wr... few more Recover phase logs It worked fine for a day and started throwing this message: 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF ps=[-A--] 2018-10-28 12:27:01 [Broker] warning Exchange cannot deliver to queue : Queue : MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 2018-10-28 12:27:01 [Broker] error Connection exception: framing-error: Queue : MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 2018-10-28 12:27:01 [Protocol] error Connection qpid.server-ip:5672-client-ip:44457 closed by error: Queue : MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete.
Re: qpid-cpp-0.35 errors
Any help in understand why/when broker throws those errors and stop receiving message would be appreciated. Not sure if any kernel tuning or broker tuning needs to be done to solve this issue. Thanks in advance, Ram On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Also from this log message (store level) it seems like waiting for AIO to > complete. > > 2018-10-28 12:27:01 [Store] critical Linear Store: Journal " name>": get_events() returned JERR_JCNTL_AIOCMPLWAIT; > wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF > ps=[-A--] > > page_state ps=[-A--] where A is AIO_PENDING > aer=1 _aio_evt_rem; ///< Remaining AIO events > > When there is or there are pending AIO, does broker close the connection? > is there any tuning that can be done to resolve this? > > Thanks, > Ram > > > > > On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> I was check the code and i see these lines for that AIO timeout. >> >> case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: >> if (++aio_sleep_cnt > MAX_AIO_SLEEPS) >> THROW_STORE_EXCEPTION("Timeout waiting for AIO in >> MessageStoreImpl::recoverMessages()"); >> ::usleep(AIO_SLEEP_TIME_US); >> break; >> >> And these are the defaults >> >> #define MAX_AIO_SLEEPS 10 // tot: ~1 sec >> >> #define AIO_SLEEP_TIME_US 10 // 0.01 ms >> >> >> RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is >> waiting for AIO. >> >> >> >> So does page got blocked and its waiting for page availability? >> >> >> Ram >> >> On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> >>> Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we >>> see this message >>> >>> 2018-10-27 18:58:25 [Store] warning Linear Store: Journal >>> "": Bad record alignment found at fid=0x4605b offs=0x107680 >>> (likely journal overwrite boundary); 19 filler record(s) required. >>> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >>> "": Recover phase write: Wrote filler record: fid=0x4605b >>> offs=0x107680 >>> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >>> "": Recover phase write: Wr... few more Recover phase logs >>> >>> It worked fine for a day and started throwing this message: >>> >>> 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": >>> get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 >>> po=0 aer=1 edac=TFFF ps=[-A--] >>> 2018-10-28 12:27:01 [Broker] warning Exchange cannot deliver to >>> queue : Queue : MessageStoreImpl::store() failed: >>> jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: >>> Timeout waiting for AIOs to complete. >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >>> 2018-10-28 12:27:01 [Broker] error Connection exception: framing-error: >>> Queue : MessageStoreImpl::store() failed: jexception 0x0202 >>> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for >>> AIOs to complete. >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >>> 2018-10-28 12:27:01 [Protocol] error Connection >>> qpid.server-ip:5672-client-ip:44457 closed by error: Queue : >>> MessageStoreImpl::store() failed: jexception 0x0202 >>> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for >>> AIOs to complete. >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) >>> 2018-10-28 12:27:01 [Protocol] error Connection >>> qpid.server-ip:5672-client-ip:44457 closed by error: illegal-argument: >>> Value for replyText is too large(320) >>> >>> Thanks, >>> Ram >>> >>> >>> On Mon, Nov 5, 2018 at 3:34 PM rammohan ganapavarapu < >>> rammohanga...@gmail.com> wrote: >>> No, local disk. On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim wrote: > On 05/11/18 22:58, rammohan ganapavarapu wrote: > > Gordon, > > > > We are using java client 0.28 version and qpidd-cpp 1.35 version > > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario > its > > happening but after i restart broker and if we wait for few days its > > happening again. From the above logs do you have any pointers to > check? > > Are you using NFS? > > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
Also from this log message (store level) it seems like waiting for AIO to complete. 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF ps=[-A--] page_state ps=[-A--] where A is AIO_PENDING aer=1 _aio_evt_rem; ///< Remaining AIO events When there is or there are pending AIO, does broker close the connection? is there any tuning that can be done to resolve this? Thanks, Ram On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > I was check the code and i see these lines for that AIO timeout. > > case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: > if (++aio_sleep_cnt > MAX_AIO_SLEEPS) > THROW_STORE_EXCEPTION("Timeout waiting for AIO in > MessageStoreImpl::recoverMessages()"); > ::usleep(AIO_SLEEP_TIME_US); > break; > > And these are the defaults > > #define MAX_AIO_SLEEPS 10 // tot: ~1 sec > > #define AIO_SLEEP_TIME_US 10 // 0.01 ms > > > RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is > waiting for AIO. > > > > So does page got blocked and its waiting for page availability? > > > Ram > > On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> >> Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we >> see this message >> >> 2018-10-27 18:58:25 [Store] warning Linear Store: Journal >> "": Bad record alignment found at fid=0x4605b offs=0x107680 >> (likely journal overwrite boundary); 19 filler record(s) required. >> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >> "": Recover phase write: Wrote filler record: fid=0x4605b >> offs=0x107680 >> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >> "": Recover phase write: Wr... few more Recover phase logs >> >> It worked fine for a day and started throwing this message: >> >> 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": >> get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 >> po=0 aer=1 edac=TFFF ps=[-A--] >> 2018-10-28 12:27:01 [Broker] warning Exchange cannot deliver to >> queue : Queue : MessageStoreImpl::store() failed: >> jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: >> Timeout waiting for AIOs to complete. >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >> 2018-10-28 12:27:01 [Broker] error Connection exception: framing-error: >> Queue : MessageStoreImpl::store() failed: jexception 0x0202 >> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for >> AIOs to complete. >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >> 2018-10-28 12:27:01 [Protocol] error Connection >> qpid.server-ip:5672-client-ip:44457 closed by error: Queue : >> MessageStoreImpl::store() failed: jexception 0x0202 >> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for >> AIOs to complete. >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) >> 2018-10-28 12:27:01 [Protocol] error Connection >> qpid.server-ip:5672-client-ip:44457 closed by error: illegal-argument: >> Value for replyText is too large(320) >> >> Thanks, >> Ram >> >> >> On Mon, Nov 5, 2018 at 3:34 PM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> No, local disk. >>> >>> On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim wrote: >>> On 05/11/18 22:58, rammohan ganapavarapu wrote: > Gordon, > > We are using java client 0.28 version and qpidd-cpp 1.35 version > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its > happening but after i restart broker and if we wait for few days its > happening again. From the above logs do you have any pointers to check? Are you using NFS? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
I was check the code and i see these lines for that AIO timeout. case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: if (++aio_sleep_cnt > MAX_AIO_SLEEPS) THROW_STORE_EXCEPTION("Timeout waiting for AIO in MessageStoreImpl::recoverMessages()"); ::usleep(AIO_SLEEP_TIME_US); break; And these are the defaults #define MAX_AIO_SLEEPS 10 // tot: ~1 sec #define AIO_SLEEP_TIME_US 10 // 0.01 ms RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is waiting for AIO. So does page got blocked and its waiting for page availability? Ram On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > > Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we see > this message > > 2018-10-27 18:58:25 [Store] warning Linear Store: Journal > "": Bad record alignment found at fid=0x4605b offs=0x107680 > (likely journal overwrite boundary); 19 filler record(s) required. > 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": > Recover phase write: Wrote filler record: fid=0x4605b offs=0x107680 > 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": > Recover phase write: Wr... few more Recover phase logs > > It worked fine for a day and started throwing this message: > > 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": > get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 > po=0 aer=1 edac=TFFF ps=[-A--] > 2018-10-28 12:27:01 [Broker] warning Exchange cannot deliver to > queue : Queue : MessageStoreImpl::store() failed: > jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: > Timeout waiting for AIOs to complete. > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > 2018-10-28 12:27:01 [Broker] error Connection exception: framing-error: > Queue : MessageStoreImpl::store() failed: jexception 0x0202 > jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for > AIOs to complete. > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > 2018-10-28 12:27:01 [Protocol] error Connection > qpid.server-ip:5672-client-ip:44457 closed by error: Queue : > MessageStoreImpl::store() failed: jexception 0x0202 > jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for > AIOs to complete. > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) > 2018-10-28 12:27:01 [Protocol] error Connection > qpid.server-ip:5672-client-ip:44457 closed by error: illegal-argument: > Value for replyText is too large(320) > > Thanks, > Ram > > > On Mon, Nov 5, 2018 at 3:34 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> No, local disk. >> >> On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim wrote: >> >>> On 05/11/18 22:58, rammohan ganapavarapu wrote: >>> > Gordon, >>> > >>> > We are using java client 0.28 version and qpidd-cpp 1.35 version >>> > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its >>> > happening but after i restart broker and if we wait for few days its >>> > happening again. From the above logs do you have any pointers to check? >>> >>> Are you using NFS? >>> >>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >>> For additional commands, e-mail: users-h...@qpid.apache.org >>> >>>
Re: qpid-cpp-0.35 errors
Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we see this message 2018-10-27 18:58:25 [Store] warning Linear Store: Journal "": Bad record alignment found at fid=0x4605b offs=0x107680 (likely journal overwrite boundary); 19 filler record(s) required. 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": Recover phase write: Wrote filler record: fid=0x4605b offs=0x107680 2018-10-27 18:58:25 [Store] notice Linear Store: Journal "": Recover phase write: Wr... few more Recover phase logs It worked fine for a day and started throwing this message: 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF ps=[-A--] 2018-10-28 12:27:01 [Broker] warning Exchange cannot deliver to queue : Queue : MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 2018-10-28 12:27:01 [Broker] error Connection exception: framing-error: Queue : MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 2018-10-28 12:27:01 [Protocol] error Connection qpid.server-ip:5672-client-ip:44457 closed by error: Queue : MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) 2018-10-28 12:27:01 [Protocol] error Connection qpid.server-ip:5672-client-ip:44457 closed by error: illegal-argument: Value for replyText is too large(320) Thanks, Ram On Mon, Nov 5, 2018 at 3:34 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > No, local disk. > > On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim wrote: > >> On 05/11/18 22:58, rammohan ganapavarapu wrote: >> > Gordon, >> > >> > We are using java client 0.28 version and qpidd-cpp 1.35 version >> > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its >> > happening but after i restart broker and if we wait for few days its >> > happening again. From the above logs do you have any pointers to check? >> >> Are you using NFS? >> >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >>
Re: qpid-cpp-0.35 errors
No, local disk. On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim wrote: > On 05/11/18 22:58, rammohan ganapavarapu wrote: > > Gordon, > > > > We are using java client 0.28 version and qpidd-cpp 1.35 version > > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its > > happening but after i restart broker and if we wait for few days its > > happening again. From the above logs do you have any pointers to check? > > Are you using NFS? > > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
On 05/11/18 22:58, rammohan ganapavarapu wrote: Gordon, We are using java client 0.28 version and qpidd-cpp 1.35 version (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its happening but after i restart broker and if we wait for few days its happening again. From the above logs do you have any pointers to check? Are you using NFS? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
Gordon, We are using java client 0.28 version and qpidd-cpp 1.35 version (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario its happening but after i restart broker and if we wait for few days its happening again. From the above logs do you have any pointers to check? We are using linear store not legacy. When i do netstat -an |grep 5672, i see two ESTABLISHED connections for a client host. This is the queue config: qpid-config queues Queue NameAttributes = 1cdfd9cd-227d-4b84-9539-ab4d67ee5a1f:0.0 auto-del excl q-001 --durable --file-size=2000 --file-count=24 --max-queue-size=1073741824 --max-queue-count=100 --limit-policy=flow-to-disk --argument no-local=False q-001-dl--durable --file-size=6000 --file-count=4 --max-queue-size=52428800 --max-queue-count=10 --limit-policy=flow-to-disk --argument no-local=False Some one posted long back similar issue but dont see any solution, http://qpid.2158936.n2.nabble.com/RE-qpid-Java-client-unable-to-send-messages-td7613136.html Thanks Ram On Mon, Nov 5, 2018 at 2:32 PM Gordon Sim wrote: > On 05/11/18 20:57, rammohan ganapavarapu wrote: > > Actually there are no messages in queue, all they messages got consumed > by > > consumer. > > But it still will not enqueue any further messages? Can you reproduce > this easily? > > One other suggestion is to try with the linear store rather than the > legacy store if possible. > > > I also observe two tcp connections to each client and for this > > client only one tcp connection. Why does qpid creates two connections? > > I don't think it does. Which client and version are you using? How are > you observing the two connections? > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
On 05/11/18 20:57, rammohan ganapavarapu wrote: Actually there are no messages in queue, all they messages got consumed by consumer. But it still will not enqueue any further messages? Can you reproduce this easily? One other suggestion is to try with the linear store rather than the legacy store if possible. I also observe two tcp connections to each client and for this client only one tcp connection. Why does qpid creates two connections? I don't think it does. Which client and version are you using? How are you observing the two connections? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
I also see this in qpidd logs for broker,store and protocol. Please help me to understand what it means? why i am i getting "Timeout waiting for AIOs to complete" ? does it means some thing wrong with journal files? 2018-02-28 13:19:00 [Store] critical Journal "q-001": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=2 pc=45 po=0 aer=1 edac:TFFF ps=[--A-] wrfc: state: Active fcntl[1]: pfid=1 ws=11524 wc=11268 rs=0 rc=0 ec=6 ac=1 2018-02-28 13:19:00 [Broker] warning Exchange ex-001 cannot deliver to queue q-001: Queue q-001: MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/fliu/rpmbuild/BUILD/qpid-0.28/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1357) 2018-02-28 13:19:00 [Broker] error Connection exception: framing-error: Queue q-001: MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/fliu/rpmbuild/BUILD/qpid-0.28/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1357) 2018-02-28 13:19:00 [Protocol] error Connection qpid.1.2.3.4:5672-1.2.3.5:28188 closed by error: Queue q-001: MessageStoreImpl::store() failed: jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for AIOs to complete. (/home/fliu/rpmbuild/BUILD/qpid-0.28/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1357)(501) On Mon, Nov 5, 2018 at 12:57 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Actually there are no messages in queue, all they messages got consumed by > consumer. I also observe two tcp connections to each client and for this > client only one tcp connection. Why does qpid creates two connections? > > Ram > > On Mon, Nov 5, 2018, 11:09 AM Gordon Sim >> Can you drain the queue? >> >> - >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >>
Re: qpid-cpp-0.35 errors
Actually there are no messages in queue, all they messages got consumed by consumer. I also observe two tcp connections to each client and for this client only one tcp connection. Why does qpid creates two connections? Ram On Mon, Nov 5, 2018, 11:09 AM Gordon Sim Can you drain the queue? > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
I am using java client version 0.28 and qpid-cpp 1.35 version, and i see this error on client side(producer). 2018-11-02 00:00:46,376 IoSender - /1.2.3.4:5672 INFO o.a.q.t.n.io.IoSender - Logger.info() : Exception in thread sending to '/ 1.2.3.4:5672': java.net.SocketException: Broken pipe (Write failed) 2018-11-02 00:00:46,377 IoReceiver - /1.2.3.4:5672 ERROR o.a.q.c.AMQConnectionDelegate_0_10 - AMQConnectionDelegate_0_10.exception() : previous exception org.apache.qpid.transport.ConnectionException: java.net.SocketException: Broken pipe (Write failed) at org.apache.qpid.transport.Connection.exception(Connection.java:546) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.exception(Assembler.java:107) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.exception(InputHandler.java:199) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:217) ~[qpid-common-0.28.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: org.apache.qpid.transport.SenderException: java.net.SocketException: Broken pipe (Write failed) at org.apache.qpid.transport.network.io.IoSender.close(IoSender.java:229) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.io.IoSender.close(IoSender.java:199) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Disassembler.close(Disassembler.java:88) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.sendConnectionCloseOkAndCloseSender(ConnectionDelegate.java:82) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:74) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:40) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionClose.dispatch(ConnectionClose.java:91) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:49) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:40) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Method.delegate(Method.java:163) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Connection.received(Connection.java:392) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Connection.received(Connection.java:62) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:183) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:131) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.received(Assembler.java:102) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.received(Assembler.java:44) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:189) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:105) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:44) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) ~[qpid-common-0.28.jar:na] ... 1 common frames omitted Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_121] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[na:1.8.0_121] at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[na:1.8.0_121] at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:308) ~[qpid-common-0.28.jar:na] ... 1 common frames omitted Thanks, Ram On Sun, Nov 4, 2018 at 7:50 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Hi, > > Any one else saw this error before? after this error broker stop taking > any messages, not sure what is causing this error. > > Thanks, > Ram > > On Fri, Nov 2, 2018 at 4:24 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Kim/Gordon, >> >> After this message broker is not accepting any more messages and keep >> throwing this message. >> >> Thanks, >> Ram >> >> On Fri, Nov 2, 2018 at 8:59 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> Any help in understating this error message would be appreciated. >>> >>> Ram >>> >>> On Wed, Oct 31, 2018 at 5:47 AM rammohan ganapavarapu < >>> rammohanga...@gmail.com> wrote: >>> Kim, Any idea about this error? Thanks, Ram On Tue, Oct 30, 2018, 2:13 PM Gordon Sim wrote: > On 30/10/18 18:59, rammohan ganapavarapu wrote: > > There are two more error from my original post, can some one help me > to > > understand
Re: qpid-cpp-0.35 errors
Hi, Any one else saw this error before? after this error broker stop taking any messages, not sure what is causing this error. Thanks, Ram On Fri, Nov 2, 2018 at 4:24 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Kim/Gordon, > > After this message broker is not accepting any more messages and keep > throwing this message. > > Thanks, > Ram > > On Fri, Nov 2, 2018 at 8:59 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Any help in understating this error message would be appreciated. >> >> Ram >> >> On Wed, Oct 31, 2018 at 5:47 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> Kim, >>> >>> Any idea about this error? >>> >>> Thanks, >>> Ram >>> >>> On Tue, Oct 30, 2018, 2:13 PM Gordon Sim wrote: >>> On 30/10/18 18:59, rammohan ganapavarapu wrote: > There are two more error from my original post, can some one help me to > understand when qpid throws these error? > > > 1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception: > not-attached: Channel 0 is not attached > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) The one above is comon when you are sending asynchronously, and a previous message caused the session to be ended with an exception frame. Any subsequent messages that were sent before the client received the exception frame result in above error. > 2. 2018-10-30 14:30:36 [Broker] error Connection exception: > framing-error: Queue ax-q-axgroup-001-consumer-group-001: > MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned > partly completed (state ENQ_PART). (This data_tok: id=1714315 state=NONE) > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > 3. 2018-10-30 14:30:36 [Protocol] error Connection > qpid.10.68.94.134:5672-10.68.94.127:39458 closed by error: Queue > ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() failed: > jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new > dtok when previous enqueue returned partly completed (state ENQ_PART). > (This data_tok: id=1714315 state=NONE) > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) Not sure what the 'partly completed state' means here. Kim, any thoughts? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
Kim/Gordon, After this message broker is not accepting any more messages and keep throwing this message. Thanks, Ram On Fri, Nov 2, 2018 at 8:59 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Any help in understating this error message would be appreciated. > > Ram > > On Wed, Oct 31, 2018 at 5:47 AM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> Kim, >> >> Any idea about this error? >> >> Thanks, >> Ram >> >> On Tue, Oct 30, 2018, 2:13 PM Gordon Sim wrote: >> >>> On 30/10/18 18:59, rammohan ganapavarapu wrote: >>> > There are two more error from my original post, can some one help me to >>> > understand when qpid throws these error? >>> > >>> > >>> > 1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception: >>> > not-attached: Channel 0 is not attached >>> > >>> >>> (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) >>> >>> The one above is comon when you are sending asynchronously, and a >>> previous message caused the session to be ended with an exception frame. >>> Any subsequent messages that were sent before the client received the >>> exception frame result in above error. >>> >>> > 2. 2018-10-30 14:30:36 [Broker] error Connection exception: >>> > framing-error: Queue ax-q-axgroup-001-consumer-group-001: >>> > MessageStoreImpl::store() failed: jexception 0x0803 >>> wmgr::enqueue() threw >>> > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue >>> returned >>> > partly completed (state ENQ_PART). (This data_tok: id=1714315 >>> state=NONE) >>> > >>> >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >>> > 3. 2018-10-30 14:30:36 [Protocol] error Connection >>> > qpid.10.68.94.134:5672-10.68.94.127:39458 closed by error: Queue >>> > ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() >>> failed: >>> > jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: >>> Enqueued new >>> > dtok when previous enqueue returned partly completed (state >>> ENQ_PART). >>> > (This data_tok: id=1714315 state=NONE) >>> > >>> >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) >>> >>> Not sure what the 'partly completed state' means here. Kim, any thoughts? >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >>> For additional commands, e-mail: users-h...@qpid.apache.org >>> >>>
Re: qpid-cpp-0.35 errors
Any help in understating this error message would be appreciated. Ram On Wed, Oct 31, 2018 at 5:47 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Kim, > > Any idea about this error? > > Thanks, > Ram > > On Tue, Oct 30, 2018, 2:13 PM Gordon Sim wrote: > >> On 30/10/18 18:59, rammohan ganapavarapu wrote: >> > There are two more error from my original post, can some one help me to >> > understand when qpid throws these error? >> > >> > >> > 1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception: >> > not-attached: Channel 0 is not attached >> > >> >> (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) >> >> The one above is comon when you are sending asynchronously, and a >> previous message caused the session to be ended with an exception frame. >> Any subsequent messages that were sent before the client received the >> exception frame result in above error. >> >> > 2. 2018-10-30 14:30:36 [Broker] error Connection exception: >> > framing-error: Queue ax-q-axgroup-001-consumer-group-001: >> > MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() >> threw >> > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue >> returned >> > partly completed (state ENQ_PART). (This data_tok: id=1714315 >> state=NONE) >> > >> >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >> > 3. 2018-10-30 14:30:36 [Protocol] error Connection >> > qpid.10.68.94.134:5672-10.68.94.127:39458 closed by error: Queue >> > ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() >> failed: >> > jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: >> Enqueued new >> > dtok when previous enqueue returned partly completed (state >> ENQ_PART). >> > (This data_tok: id=1714315 state=NONE) >> > >> >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) >> >> Not sure what the 'partly completed state' means here. Kim, any thoughts? >> >> - >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org >> For additional commands, e-mail: users-h...@qpid.apache.org >> >>
Re: qpid-cpp-0.35 errors
Kim, Any idea about this error? Thanks, Ram On Tue, Oct 30, 2018, 2:13 PM Gordon Sim wrote: > On 30/10/18 18:59, rammohan ganapavarapu wrote: > > There are two more error from my original post, can some one help me to > > understand when qpid throws these error? > > > > > > 1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception: > > not-attached: Channel 0 is not attached > > > > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) > > The one above is comon when you are sending asynchronously, and a > previous message caused the session to be ended with an exception frame. > Any subsequent messages that were sent before the client received the > exception frame result in above error. > > > 2. 2018-10-30 14:30:36 [Broker] error Connection exception: > > framing-error: Queue ax-q-axgroup-001-consumer-group-001: > > MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() > threw > > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue > returned > > partly completed (state ENQ_PART). (This data_tok: id=1714315 > state=NONE) > > > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > > 3. 2018-10-30 14:30:36 [Protocol] error Connection > > qpid.10.68.94.134:5672-10.68.94.127:39458 closed by error: Queue > > ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() > failed: > > jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: > Enqueued new > > dtok when previous enqueue returned partly completed (state > ENQ_PART). > > (This data_tok: id=1714315 state=NONE) > > > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) > > Not sure what the 'partly completed state' means here. Kim, any thoughts? > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
Gordon, Thank you. On Tue, Oct 30, 2018 at 2:13 PM Gordon Sim wrote: > On 30/10/18 18:59, rammohan ganapavarapu wrote: > > There are two more error from my original post, can some one help me to > > understand when qpid throws these error? > > > > > > 1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception: > > not-attached: Channel 0 is not attached > > > > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) > > The one above is comon when you are sending asynchronously, and a > previous message caused the session to be ended with an exception frame. > Any subsequent messages that were sent before the client received the > exception frame result in above error. > > > 2. 2018-10-30 14:30:36 [Broker] error Connection exception: > > framing-error: Queue ax-q-axgroup-001-consumer-group-001: > > MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() > threw > > JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue > returned > > partly completed (state ENQ_PART). (This data_tok: id=1714315 > state=NONE) > > > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > > 3. 2018-10-30 14:30:36 [Protocol] error Connection > > qpid.10.68.94.134:5672-10.68.94.127:39458 closed by error: Queue > > ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() > failed: > > jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: > Enqueued new > > dtok when previous enqueue returned partly completed (state > ENQ_PART). > > (This data_tok: id=1714315 state=NONE) > > > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) > > Not sure what the 'partly completed state' means here. Kim, any thoughts? > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
On 30/10/18 18:59, rammohan ganapavarapu wrote: There are two more error from my original post, can some one help me to understand when qpid throws these error? 1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: Channel 0 is not attached (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) The one above is comon when you are sending asynchronously, and a previous message caused the session to be ended with an exception frame. Any subsequent messages that were sent before the client received the exception frame result in above error. 2. 2018-10-30 14:30:36 [Broker] error Connection exception: framing-error: Queue ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=1714315 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 3. 2018-10-30 14:30:36 [Protocol] error Connection qpid.10.68.94.134:5672-10.68.94.127:39458 closed by error: Queue ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=1714315 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) Not sure what the 'partly completed state' means here. Kim, any thoughts? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
Just to add to this , the store level error corresponding to those above errors is : 2018-02-28 13:19:00 [Store] critical Journal "ax-q-axgroup-001-consumer-group-001": get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=2 pc=45 po=0 aer=1 edac:TFFF ps=[--A-] wrfc: state: Active fcntl[1]: pfid=1 ws=11524 wc=11268 rs=0 rc=0 ec=6 ac=1 Thanks, Ram On Mon, Oct 22, 2018 at 2:20 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Hi, > > I am seeing lot of these messages in my qpidd logs and i am not sure why > am i seeing these, can some one explain? > > 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver > to queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on > prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: > 100, size: 1073741824] > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > 2018-10-22 07:39:02 [Broker] error Execution exception: > resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: > current=[count: 12567, size: 1073741816], max=[count: 100, size: > 1073741824] > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: > Channel 0 is not attached > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) > > > 2018-10-22 07:37:02 [Broker] error Connection exception: framing-error: > Queue prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 > wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous > enqueue returned partly completed (state ENQ_PART). (This data_tok: > id=4682049 state=NONE) > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > 2018-10-22 07:37:02 [Protocol] error Connection > qpid.10.68.94.117:5672-10.66.244.23:46574 closed by error: Queue > prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 > wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous > enqueue returned partly completed (state ENQ_PART). (This data_tok: > id=4682049 state=NONE) > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) > 2018-10-22 07:37:02 [Protocol] error Connection > qpid.10.68.94.117:5672-10.66.244.23:46574 closed by error: > illegal-argument: Value for replyText is too large(320) > 2018-10-22 07:37:05 [Broker] notice Broker (pid=35293) shut-down > > > Thanks, > Ram > >
Re: qpid-cpp-0.35 errors
There are two more error from my original post, can some one help me to understand when qpid throws these error? 1. 1. 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: Channel 0 is not attached (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) 2. 2018-10-30 14:30:36 [Broker] error Connection exception: framing-error: Queue ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=1714315 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 3. 2018-10-30 14:30:36 [Protocol] error Connection qpid.10.68.94.134:5672-10.68.94.127:39458 closed by error: Queue ax-q-axgroup-001-consumer-group-001: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=1714315 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) Thanks, Ram On Mon, Oct 29, 2018 at 2:48 PM Gordon Sim wrote: > On 29/10/18 21:45, rammohan ganapavarapu wrote: > > I will try to produce, do have any suggestions on what to check in trace > > logs? > > for qpidd, adding --log-enable trace+:Network to whatever other logging > options you have should do it > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: qpid-cpp-0.35 errors
On 29/10/18 21:45, rammohan ganapavarapu wrote: I will try to produce, do have any suggestions on what to check in trace logs? for qpidd, adding --log-enable trace+:Network to whatever other logging options you have should do it - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-cpp-0.35 errors
I will try to produce, do have any suggestions on what to check in trace logs? On Fri, Oct 26, 2018 at 2:15 AM Gordon Sim wrote: > On 26/10/18 00:44, rammohan ganapavarapu wrote: > > Any idea when do we see this error? > > In AMQP 0-10, the connection-close frame allows an informational message > (replyText) to be included. This must be less than 256 bytes. It sounds > like something is either trying to exceed that, or succeeding and > violating the protocol. > > If you can reproduce with a protocol trace, that will speed up resolution. > > > > > On Tue, Oct 23, 2018 at 10:42 AM rammohan ganapavarapu < > > rammohanga...@gmail.com> wrote: > > > >> That error i get it but what is this error i am seeing in client ? > >> > >> > >> 2018-10-23 17:10:57,997 IoReceiver - /10.68.94.134:5672 ERROR > >> o.a.q.c.AMQConnectionDelegate_0_10 - > AMQConnectionDelegate_0_10.closed() : > >> connection exception: conn:5668fdd1 > >> org.apache.qpid.transport.ConnectionException: illegal-argument: Value > for > >> replyText is too large > >> at org.apache.qpid.transport.Connection.closeCode(Connection.java:556) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:75) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:40) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.ConnectionClose.dispatch(ConnectionClose.java:91) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:49) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:40) > >> ~[qpid-common-0.28.jar:na] > >> at org.apache.qpid.transport.Method.delegate(Method.java:163) > >> ~[qpid-common-0.28.jar:na] > >> at org.apache.qpid.transport.Connection.received(Connection.java:392) > >> ~[qpid-common-0.28.jar:na] > >> at org.apache.qpid.transport.Connection.received(Connection.java:62) > >> ~[qpid-common-0.28.jar:na] > >> at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) > >> ~[qpid-common-0.28.jar:na] > >> at > >> org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:183) > >> ~[qpid-common-0.28.jar:na] > >> at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:131) > >> ~[qpid-common-0.28.jar:na] > >> at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) > >> ~[qpid-common-0.28.jar:na] > >> at > >> org.apache.qpid.transport.network.Assembler.received(Assembler.java:102) > >> ~[qpid-common-0.28.jar:na] > >> at > org.apache.qpid.transport.network.Assembler.received(Assembler.java:44) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:189) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:105) > >> ~[qpid-common-0.28.jar:na] > >> at > >> > org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:44) > >> ~[qpid-common-0.28.jar:na] > >> at > >> org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) > >> ~[qpid-common-0.28.jar:na] > >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > >> > >> Ram > >> > >> On Mon, Oct 22, 2018 at 3:41 PM Chester wrote: > >> > >>> 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot > deliver to > >>> > >>> queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on > >>> > >>> prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: > >>> > >>> 100, size: 1073741824] > >>> > >>> (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > >>> > >>> > >>> Looks like you're hitting a queue size (bytes) limit of ~1GB. See the > >>> Broker Book for the configuration option that sets this limit [1]. > >>> > >>> [1] > >>> > >>> > https://qpid.apache.org/releases/qpid-cpp-1.38.0/cpp-broker/book/ch01s02.html#CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints > >>> > >>> > >>> On Mon, Oct 22, 2018 at 5:21 PM rammohan ganapavarapu < > >>> rammohanga...@gmail.com> wrote: > >>> > Hi, > > I am seeing lot of these messages in my qpidd logs and i am not sure > >>> why am > i seeing these, can some one explain? > > 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot > deliver > >>> to > queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded > on > prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: > 100, size: 1073741824] > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > 2018-10-22 07:39:02 [Broker] error Execution exception: > resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: > current=[count: 12567, size: 1073741816], max=[count: 100, size: > 1073741824] >
Re: qpid-cpp-0.35 errors
On 26/10/18 00:44, rammohan ganapavarapu wrote: Any idea when do we see this error? In AMQP 0-10, the connection-close frame allows an informational message (replyText) to be included. This must be less than 256 bytes. It sounds like something is either trying to exceed that, or succeeding and violating the protocol. If you can reproduce with a protocol trace, that will speed up resolution. On Tue, Oct 23, 2018 at 10:42 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: That error i get it but what is this error i am seeing in client ? 2018-10-23 17:10:57,997 IoReceiver - /10.68.94.134:5672 ERROR o.a.q.c.AMQConnectionDelegate_0_10 - AMQConnectionDelegate_0_10.closed() : connection exception: conn:5668fdd1 org.apache.qpid.transport.ConnectionException: illegal-argument: Value for replyText is too large at org.apache.qpid.transport.Connection.closeCode(Connection.java:556) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:75) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:40) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionClose.dispatch(ConnectionClose.java:91) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:49) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:40) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Method.delegate(Method.java:163) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Connection.received(Connection.java:392) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Connection.received(Connection.java:62) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:183) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:131) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.received(Assembler.java:102) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.received(Assembler.java:44) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:189) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:105) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:44) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) ~[qpid-common-0.28.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Ram On Mon, Oct 22, 2018 at 3:41 PM Chester wrote: 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: 100, size: 1073741824] (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) Looks like you're hitting a queue size (bytes) limit of ~1GB. See the Broker Book for the configuration option that sets this limit [1]. [1] https://qpid.apache.org/releases/qpid-cpp-1.38.0/cpp-broker/book/ch01s02.html#CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints On Mon, Oct 22, 2018 at 5:21 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: Hi, I am seeing lot of these messages in my qpidd logs and i am not sure why am i seeing these, can some one explain? 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: 100, size: 1073741824] (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) 2018-10-22 07:39:02 [Broker] error Execution exception: resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: 100, size: 1073741824] (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: Channel 0 is not attached (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) 2018-10-22 07:37:02 [Broker] error Connection exception: framing-error: Queue prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=4682049 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 2018-10-22 07:37:02 [Protocol] error Connection
Re: qpid-cpp-0.35 errors
Any idea when do we see this error? On Tue, Oct 23, 2018 at 10:42 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > That error i get it but what is this error i am seeing in client ? > > > 2018-10-23 17:10:57,997 IoReceiver - /10.68.94.134:5672 ERROR > o.a.q.c.AMQConnectionDelegate_0_10 - AMQConnectionDelegate_0_10.closed() : > connection exception: conn:5668fdd1 > org.apache.qpid.transport.ConnectionException: illegal-argument: Value for > replyText is too large > at org.apache.qpid.transport.Connection.closeCode(Connection.java:556) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:75) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:40) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.ConnectionClose.dispatch(ConnectionClose.java:91) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:49) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:40) > ~[qpid-common-0.28.jar:na] > at org.apache.qpid.transport.Method.delegate(Method.java:163) > ~[qpid-common-0.28.jar:na] > at org.apache.qpid.transport.Connection.received(Connection.java:392) > ~[qpid-common-0.28.jar:na] > at org.apache.qpid.transport.Connection.received(Connection.java:62) > ~[qpid-common-0.28.jar:na] > at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:183) > ~[qpid-common-0.28.jar:na] > at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:131) > ~[qpid-common-0.28.jar:na] > at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.network.Assembler.received(Assembler.java:102) > ~[qpid-common-0.28.jar:na] > at org.apache.qpid.transport.network.Assembler.received(Assembler.java:44) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:189) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:105) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:44) > ~[qpid-common-0.28.jar:na] > at > org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) > ~[qpid-common-0.28.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > > Ram > > On Mon, Oct 22, 2018 at 3:41 PM Chester wrote: > >> 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to >> >> queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on >> >> prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: >> >> 100, size: 1073741824] >> >> (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) >> >> >> Looks like you're hitting a queue size (bytes) limit of ~1GB. See the >> Broker Book for the configuration option that sets this limit [1]. >> >> [1] >> >> https://qpid.apache.org/releases/qpid-cpp-1.38.0/cpp-broker/book/ch01s02.html#CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints >> >> >> On Mon, Oct 22, 2018 at 5:21 PM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >> > Hi, >> > >> > I am seeing lot of these messages in my qpidd logs and i am not sure >> why am >> > i seeing these, can some one explain? >> > >> > 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver >> to >> > queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on >> > prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: >> > 100, size: 1073741824] >> > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) >> > >> > 2018-10-22 07:39:02 [Broker] error Execution exception: >> > resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: >> > current=[count: 12567, size: 1073741816], max=[count: 100, size: >> > 1073741824] >> > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) >> > >> > 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: >> Channel >> > 0 is not attached >> > >> > >> (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) >> > >> > >> > 2018-10-22 07:37:02 [Broker] error Connection exception: framing-error: >> > Queue prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 >> > wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when >> previous >> > enqueue returned partly completed (state ENQ_PART). (This data_tok: >> > id=4682049 state=NONE) >> > >> > >> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >> > 2018-10-22 07:37:02 [Protocol] error Connection >> > qpid.10.68.94.117:5672-10.66.244.23:46574 closed by
Re: qpid-cpp-0.35 errors
That error i get it but what is this error i am seeing in client ? 2018-10-23 17:10:57,997 IoReceiver - /10.68.94.134:5672 ERROR o.a.q.c.AMQConnectionDelegate_0_10 - AMQConnectionDelegate_0_10.closed() : connection exception: conn:5668fdd1 org.apache.qpid.transport.ConnectionException: illegal-argument: Value for replyText is too large at org.apache.qpid.transport.Connection.closeCode(Connection.java:556) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:75) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.connectionClose(ConnectionDelegate.java:40) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionClose.dispatch(ConnectionClose.java:91) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:49) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:40) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Method.delegate(Method.java:163) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Connection.received(Connection.java:392) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.Connection.received(Connection.java:62) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:183) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:131) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.received(Assembler.java:102) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.Assembler.received(Assembler.java:44) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:189) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:105) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:44) ~[qpid-common-0.28.jar:na] at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) ~[qpid-common-0.28.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Ram On Mon, Oct 22, 2018 at 3:41 PM Chester wrote: > 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to > > queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on > > prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: > > 100, size: 1073741824] > > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > > Looks like you're hitting a queue size (bytes) limit of ~1GB. See the > Broker Book for the configuration option that sets this limit [1]. > > [1] > > https://qpid.apache.org/releases/qpid-cpp-1.38.0/cpp-broker/book/ch01s02.html#CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints > > > On Mon, Oct 22, 2018 at 5:21 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > > > Hi, > > > > I am seeing lot of these messages in my qpidd logs and i am not sure why > am > > i seeing these, can some one explain? > > > > 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver > to > > queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on > > prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: > > 100, size: 1073741824] > > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > > > 2018-10-22 07:39:02 [Broker] error Execution exception: > > resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: > > current=[count: 12567, size: 1073741816], max=[count: 100, size: > > 1073741824] > > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > > > 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: > Channel > > 0 is not attached > > > > > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) > > > > > > 2018-10-22 07:37:02 [Broker] error Connection exception: framing-error: > > Queue prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 > > wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when > previous > > enqueue returned partly completed (state ENQ_PART). (This data_tok: > > id=4682049 state=NONE) > > > > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > > 2018-10-22 07:37:02 [Protocol] error Connection > > qpid.10.68.94.117:5672-10.66.244.23:46574 closed by error: Queue > > prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 > > wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when > previous > > enqueue returned partly completed (state ENQ_PART). (This data_tok: > > id=4682049 state=NONE) > > > > >
Re: qpid-cpp-0.35 errors
2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: 100, size: 1073741824] (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) Looks like you're hitting a queue size (bytes) limit of ~1GB. See the Broker Book for the configuration option that sets this limit [1]. [1] https://qpid.apache.org/releases/qpid-cpp-1.38.0/cpp-broker/book/ch01s02.html#CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints On Mon, Oct 22, 2018 at 5:21 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Hi, > > I am seeing lot of these messages in my qpidd logs and i am not sure why am > i seeing these, can some one explain? > > 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to > queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on > prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: > 100, size: 1073741824] > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > 2018-10-22 07:39:02 [Broker] error Execution exception: > resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: > current=[count: 12567, size: 1073741816], max=[count: 100, size: > 1073741824] > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) > > 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: Channel > 0 is not attached > > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) > > > 2018-10-22 07:37:02 [Broker] error Connection exception: framing-error: > Queue prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 > wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous > enqueue returned partly completed (state ENQ_PART). (This data_tok: > id=4682049 state=NONE) > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) > 2018-10-22 07:37:02 [Protocol] error Connection > qpid.10.68.94.117:5672-10.66.244.23:46574 closed by error: Queue > prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 > wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous > enqueue returned partly completed (state ENQ_PART). (This data_tok: > id=4682049 state=NONE) > > (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) > 2018-10-22 07:37:02 [Protocol] error Connection > qpid.10.68.94.117:5672-10.66.244.23:46574 closed by error: > illegal-argument: Value for replyText is too large(320) > 2018-10-22 07:37:05 [Broker] notice Broker (pid=35293) shut-down > > > Thanks, > Ram >
qpid-cpp-0.35 errors
Hi, I am seeing lot of these messages in my qpidd logs and i am not sure why am i seeing these, can some one explain? 2018-10-22 07:39:02 [Broker] warning Exchange ex-group-1 cannot deliver to queue prod-queue-01: resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: 100, size: 1073741824] (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) 2018-10-22 07:39:02 [Broker] error Execution exception: resource-limit-exceeded: Maximum depth exceeded on prod-queue-01: current=[count: 12567, size: 1073741816], max=[count: 100, size: 1073741824] (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662) 2018-10-22 08:05:30 [Broker] error Channel exception: not-attached: Channel 0 is not attached (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/amqp_0_10/SessionHandler.cpp:39) 2018-10-22 07:37:02 [Broker] error Connection exception: framing-error: Queue prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=4682049 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) 2018-10-22 07:37:02 [Protocol] error Connection qpid.10.68.94.117:5672-10.66.244.23:46574 closed by error: Queue prod-queue-01: MessageStoreImpl::store() failed: jexception 0x0803 wmgr::enqueue() threw JERR_WMGR_ENQDISCONT: Enqueued new dtok when previous enqueue returned partly completed (state ENQ_PART). (This data_tok: id=4682049 state=NONE) (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) 2018-10-22 07:37:02 [Protocol] error Connection qpid.10.68.94.117:5672-10.66.244.23:46574 closed by error: illegal-argument: Value for replyText is too large(320) 2018-10-22 07:37:05 [Broker] notice Broker (pid=35293) shut-down Thanks, Ram