Any help in understand why/when broker throws those errors and stop receiving message would be appreciated.
Not sure if any kernel tuning or broker tuning needs to be done to solve this issue. Thanks in advance, Ram On Tue, Nov 6, 2018 at 8:35 AM rammohan ganapavarapu < [email protected]> wrote: > Also from this log message (store level) it seems like waiting for AIO to > complete. > > 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "<journal > name>": get_events() returned JERR_JCNTL_AIOCMPLWAIT; > wmgr_status: wmgr: pi=25 pc=8 po=0 aer=1 edac=TFFF > ps=[-------------------------A------] > > page_state ps=[-------------------------A------] where A is AIO_PENDING > aer=1 _aio_evt_rem; ///< Remaining AIO events > > When there is or there are pending AIO, does broker close the connection? > is there any tuning that can be done to resolve this? > > Thanks, > Ram > > > > > On Mon, Nov 5, 2018 at 8:55 PM rammohan ganapavarapu < > [email protected]> wrote: > >> I was check the code and i see these lines for that AIO timeout. >> >> case qpid::linearstore::journal::RHM_IORES_PAGE_AIOWAIT: >> if (++aio_sleep_cnt > MAX_AIO_SLEEPS) >> THROW_STORE_EXCEPTION("Timeout waiting for AIO in >> MessageStoreImpl::recoverMessages()"); >> ::usleep(AIO_SLEEP_TIME_US); >> break; >> >> And these are the defaults >> >> #define MAX_AIO_SLEEPS 100000 // tot: ~1 sec >> >> #define AIO_SLEEP_TIME_US 10 // 0.01 ms >> >> >> RHM_IORES_PAGE_AIOWAIT, ///< IO operation suspended - next page is >> waiting for AIO. >> >> >> >> So does page got blocked and its waiting for page availability? >> >> >> Ram >> >> On Mon, Nov 5, 2018 at 8:00 PM rammohan ganapavarapu < >> [email protected]> wrote: >> >>> >>> Actually we have upgraded from qpid-cpp 0.28 to 1.35 and after that we >>> see this message >>> >>> 2018-10-27 18:58:25 [Store] warning Linear Store: Journal >>> "<journal-name>": Bad record alignment found at fid=0x4605b offs=0x107680 >>> (likely journal overwrite boundary); 19 filler record(s) required. >>> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >>> "<journal-name>": Recover phase write: Wrote filler record: fid=0x4605b >>> offs=0x107680 >>> 2018-10-27 18:58:25 [Store] notice Linear Store: Journal >>> "<journal-name>": Recover phase write: Wr... few more Recover phase logs >>> >>> It worked fine for a day and started throwing this message: >>> >>> 2018-10-28 12:27:01 [Store] critical Linear Store: Journal "<name>": >>> get_events() returned JERR_JCNTL_AIOCMPLWAIT; wmgr_status: wmgr: pi=25 pc=8 >>> po=0 aer=1 edac=TFFF ps=[-------------------------A------] >>> 2018-10-28 12:27:01 [Broker] warning Exchange <name> cannot deliver to >>> queue <queue-name>: Queue <queue-name>: MessageStoreImpl::store() failed: >>> jexception 0x0202 jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: >>> Timeout waiting for AIOs to complete. >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >>> 2018-10-28 12:27:01 [Broker] error Connection exception: framing-error: >>> Queue <queue-name>: MessageStoreImpl::store() failed: jexception 0x0202 >>> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for >>> AIOs to complete. >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211) >>> 2018-10-28 12:27:01 [Protocol] error Connection >>> qpid.server-ip:5672-client-ip:44457 closed by error: Queue <queue-name>: >>> MessageStoreImpl::store() failed: jexception 0x0202 >>> jcntl::handle_aio_wait() threw JERR_JCNTL_AIOCMPLWAIT: Timeout waiting for >>> AIOs to complete. >>> (/home/rganapavarapu/rpmbuild/BUILD/qpid-cpp-1.35.0/src/qpid/linearstore/MessageStoreImpl.cpp:1211)(501) >>> 2018-10-28 12:27:01 [Protocol] error Connection >>> qpid.server-ip:5672-client-ip:44457 closed by error: illegal-argument: >>> Value for replyText is too large(320) >>> >>> Thanks, >>> Ram >>> >>> >>> On Mon, Nov 5, 2018 at 3:34 PM rammohan ganapavarapu < >>> [email protected]> wrote: >>> >>>> No, local disk. >>>> >>>> On Mon, Nov 5, 2018 at 3:26 PM Gordon Sim <[email protected]> wrote: >>>> >>>>> On 05/11/18 22:58, rammohan ganapavarapu wrote: >>>>> > Gordon, >>>>> > >>>>> > We are using java client 0.28 version and qpidd-cpp 1.35 version >>>>> > (qpid-cpp-server-1.35.0-1.el7.x86_64), i dont know at what scenario >>>>> its >>>>> > happening but after i restart broker and if we wait for few days its >>>>> > happening again. From the above logs do you have any pointers to >>>>> check? >>>>> >>>>> Are you using NFS? >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>>
