Ekta, Either there's a not-yet discovered (or fixed since 1.16) software defect causing this problem that you are seeing or there's some kind of misconfiguration in your setup. If the software requires patching to fix a bug, you will need to go through an upgrade to deploy it.
I would suggest looking at the logs from the router(s) that are involved in the issues you are seeing to find some clues as to what is going on. For example, look for the string "Auto Link Activated" in the logs. This is logged every time the router sets up a producer or consumer on a broker connection. You should see exactly one of these for each auto-link you have configured whenever a new broker connection is established. This should shed some light on whether there's a configuration issue or a code issue. -Ted On Wed, Jan 11, 2023 at 1:35 PM Ekta Awasthi <ekta.awas...@theodpcorp.com> wrote: > Hello Ted, > > I agree with you, we can certainly try upgrading but since this issue is > happening *directly* in *PROD* and so *frequently*, we need a quick > solution to solve these issues. Any help/suggestions will be greatly > appreciated. > > *Ekta Awasthi*, > > Engineer, EAI Operations & Support | Office Depot, Inc. > 6600 North Military Trail | Boca Raton, FL 33496-2434 > Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awas...@officedepot.com > > > > *-- Tips for EAI Support Engagement --* > > -EAI Pre-Prod Support: Create requests on the following JIRA board EAI > Operations Support > <https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS> > > -EAI Production Support: Create requests via IT Service Desk > <https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service > portal, instructions click here > <https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: > EAI Support queue --> ODP - Enterprise Apps Integration Support > > -As a reminder, the Service Availability Managers should be engaged for > any service impacting issues, with a ***Page*** to > naitavailabilitym...@officedepot.com or by initiating a MIRT > > ------------------------------ > *From:* Ted Ross <tr...@redhat.com> > *Sent:* Friday, January 6, 2023 3:37 PM > *To:* Ekta Awasthi <ekta.awas...@theodpcorp.com> > *Cc:* users@qpid.apache.org <users@qpid.apache.org>; Ajit Tathawade > (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale (Contractor) < > nilesh.khok...@officedepot.com>; EAIOpsSupport < > eaiopssupp...@officedepot.com> > *Subject:* Re: More than configured consumers count > > [CAUTION: EXTERNAL SENDER] > > Ekta, > > One thing you should consider is to bring your Qpid Dispatch Router code > up to the latest version. You are running 1.16.0 which is pretty old. > Many bugs have been fixed since 1.16.0. > > -Ted > > > On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <ekta.awas...@theodpcorp.com> > wrote: > > Hello Ted, > > Regarding Are the od-broker-1-[ms] connections configured via connectors > or listeners? ---> They are configured via connectors only below is an > example. > > #od-broker-1-m connector > connector{ > name: od-broker-1-m > host: activemq-test-1.odprivatecloud.com > port: 61618 > role: route-container > linkCapacity: 40 > sslProfile: od-router-2-test-ssl-profile > verifyHostName: no #Since we have cert/hostname mismatches > #saslMechanisms: PLAIN > } > > *Ekta Awasthi*, > > ------------------------------ > *From:* Ted Ross <tr...@redhat.com> > *Sent:* Tuesday, January 3, 2023 10:43 AM > *To:* Ekta Awasthi <ekta.awas...@theodpcorp.com> > *Cc:* users@qpid.apache.org <users@qpid.apache.org>; Ajit Tathawade > (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale (Contractor) < > nilesh.khok...@officedepot.com>; EAIOpsSupport < > eaiopssupp...@officedepot.com> > *Subject:* Re: More than configured consumers count > > [CAUTION: EXTERNAL SENDER] > > Another question... > > Are the od-broker-1-[ms] connections configured via connectors or > listeners? > > -Ted > > > On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote: > > Happy New Year Ekta, > > Are your micro-services message producers or consumers? > > When you see the more-than-expected consumers in Hawtio, get the link > status from the routers using "qdstat -l". There _should_ be one link for > each auto-link as long as the targeted broker is reachable. This should > provide some clue as to what is happening. > > -Ted > > > On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <ekta.awas...@theodpcorp.com> > wrote: > > Hello Ted, > > Thank you for your response. > > We are using AutoLinks for our addresses. Below is an example of a > autolink queue in qdrouterd.conf for one activemq broker pair. Thanks > > address { > prefix: test-queue > waypoint: yes > } > > autoLink { > connection: od-broker-1-m > addr: test-queue > dir: in > } > > autoLink { > connection: od-broker-1-m > addr: test-queue > dir: out > } > > autoLink { > connection: od-broker-1-s > addr: test-queue > dir: in > } > > autoLink { > connection: od-broker-1-s > addr: test-queue > dir: out > } > > *Ekta Awasthi*, > > ------------------------------ > *From:* Ted Ross <tr...@redhat.com> > *Sent:* Tuesday, January 3, 2023 8:47 AM > *To:* users@qpid.apache.org <users@qpid.apache.org> > *Cc:* Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh > Khokale (Contractor) <nilesh.khok...@officedepot.com>; EAIOpsSupport < > eaiopssupp...@officedepot.com> > *Subject:* Re: More than configured consumers count > > [CAUTION: EXTERNAL SENDER] > > Hi Etka, > > Can you tell us how you have configured the qdrouters to act as > consumers? Are you using auto-links or are you using link-routed addresses? > > -Ted > > > On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi > <ekta.awas...@theodpcorp.com.invalid> wrote: > > Hello, > > Is this the right mailing DL for QPID related queries. > > Just wondering as I have not heard back but given it is a holiday season, > so I will sit back and wait patiently for someone to reply. > > Thanks In Advance. > Ekta Awasthi > > ________________________________ > From: Ekta Awasthi <ekta.awas...@theodpcorp.com> > Sent: Thursday, December 22, 2022 1:00 AM > To: users@qpid.apache.org <users@qpid.apache.org> > Cc: Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh > Khokale (Contractor) <nilesh.khok...@officedepot.com>; EAIOpsSupport < > eaiopssupp...@officedepot.com> > Subject: More than configured consumers count > > Hi There, > > We have few issues with the qpid-dispatch router where we are seeing > multiple consumers are getting created randomly by our qpid dispatch > routers. We are unable to trace nor able to replicate this issue in our > lower env's. Below is the architecture diagram. > > I apologize if this isn't the right place to ask this question. In case > there is another place to communicate, do let me know as this is my first > time reaching out regarding qpid query😀 Please provide any suggestions or > feedback to help us understand this issue better. Thanks > > Flow: > Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch > Routers acting up as consumers in front of brokers (1.16 version) ---> 4 > Broker pairs of activemq(2.18 version) masters and slaves (independent > pairs) > > Problem statement # 1 > We are seeing more than the configured number of consumers count in our > activemq hawtio console causing messages to sit in delivering count which > are Un browsable since those messages are currently being delivered to its > consumers. Having only two qpid dispatch routers(acting as consumers) > infront of our activemq brokers, the count should always remain 2 but at > times it is going 3 sometimes 4. To resolve this issue, we are having to > bounce the qpid’s to release the stuck/bad consumer so that messages can be > processed/consumed. > > Problem statement # 2 > At times we see messages going to delivering count even when there are > only two configured consumers(2qpids) showing in activemq hawtio console. > We don't know the cause to why the messages get stuck in delivering count. > To resolve this, we tried restarting the consumer service but that did not > help. Next up we tried restarting the brokers, that did not help and > noticed that all the stuck delivering messages, the broker slowly replayed > back and Eventually the stuck messages came back to delivering count post > broker restart. To resolve this issue, we are having to bounce the qpid’s > to release the messages and that fixes the issue. > > Version of qpid-dispatch router > qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm > qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm > > > Ekta Awasthi > > CONFIDENTIALITY NOTICE: The information contained in this email and > attached document(s) may contain confidential information that is intended > only for the addressee(s). If you are not the intended recipient, you are > hereby advised that any disclosure, copying, distribution or the taking of > any action in reliance upon the information is prohibited. If you have > received this email in error, please immediately notify the sender and > delete it from your system. > >
