On Fri, Jan 13, 2023 at 2:21 PM Ekta Awasthi <ekta.awas...@theodpcorp.com.invalid> wrote:
> Hello Ted, > > I did go through both the qpid logs and saw below. > > I could see when the second qpid initiated the extra consumer link on > 1/9/2023 at 4.24 PM EST there were 2775 events stating, "Auto Link > deactivated" and then shortly after I could see about 5550 events stating > "Auto Link Activated" That is exactly double the links which second qpid > activated and therefore we could see two consumer link coming from qpid-2 > and 1 consumer link from qpid-1 for each queue. > > Why it activated/created double the links for all the queues is out of my > mind and that too in the middle of the day when there was no maintenance, > or anything was happening. > Deactivation happens when: A) The configuration is removed (not the case here) B) The connection to the broker is dropped Check that log to see if the deactivations are a result of a dropped connection to the broker. Similarly, the activations occur as a result of a new connection being established with the broker. Ask yourself: How many auto-links do you expect to have? How many connections do you expect to see auto-links on? Do you see multiple activations of the same auto-link on the same connection? > > Why all of the sudden it deactivated all the links it was attached to? do > we know what triggered that? > > Ekta Awasthi, > Engineer, EAI Operations & Support | Office Depot, Inc. > 6600 North Military Trail | Boca Raton, FL 33496-2434 > Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awas...@officedepot.com > <mailto:ekta.awas...@officedepot.com> > > -- Tips for EAI Support Engagement -- > -EAI Pre-Prod Support: Create requests on the following JIRA board EAI > Operations Support< > https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS > > > -EAI Production Support: Create requests via IT Service Desk< > https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service > portal, instructions click here< > https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: > EAI Support queue --> ODP - Enterprise Apps Integration Support > -As a reminder, the Service Availability Managers should be engaged for > any service impacting issues, with a ***Page*** to > naitavailabilitym...@officedepot.com<mailto: > naitavailabilitym...@officedepot.com> or by initiating a MIRT > > ________________________________ > From: Ted Ross <tr...@redhat.com> > Sent: Thursday, January 12, 2023 4:32 PM > To: Ekta Awasthi <ekta.awas...@theodpcorp.com> > Cc: users@qpid.apache.org <users@qpid.apache.org>; Ajit Tathawade > (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale (Contractor) < > nilesh.khok...@officedepot.com>; EAIOpsSupport < > eaiopssupp...@officedepot.com> > Subject: Re: More than configured consumers count > > [CAUTION: EXTERNAL SENDER] > > > Ekta, > > Either there's a not-yet discovered (or fixed since 1.16) software defect > causing this problem that you are seeing or there's some kind of > misconfiguration in your setup. If the software requires patching to fix a > bug, you will need to go through an upgrade to deploy it. > > I would suggest looking at the logs from the router(s) that are involved in > the issues you are seeing to find some clues as to what is going on. > > For example, look for the string "Auto Link Activated" in the logs. This > is logged every time the router sets up a producer or consumer on a broker > connection. You should see exactly one of these for each auto-link you > have configured whenever a new broker connection is established. This > should shed some light on whether there's a configuration issue or a code > issue. > > -Ted > > > On Wed, Jan 11, 2023 at 1:35 PM Ekta Awasthi <ekta.awas...@theodpcorp.com> > wrote: > > > Hello Ted, > > > > I agree with you, we can certainly try upgrading but since this issue is > > happening *directly* in *PROD* and so *frequently*, we need a quick > > solution to solve these issues. Any help/suggestions will be greatly > > appreciated. > > > > *Ekta Awasthi*, > > > > Engineer, EAI Operations & Support | Office Depot, Inc. > > 6600 North Military Trail | Boca Raton, FL 33496-2434 > > Office: 561-438-3552 | Mobile: 206-966-5577 | > ekta.awas...@officedepot.com > > > > > > > > *-- Tips for EAI Support Engagement --* > > > > -EAI Pre-Prod Support: Create requests on the following JIRA board EAI > > Operations Support > > < > https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS > > > > > > -EAI Production Support: Create requests via IT Service Desk > > <https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service > > portal, instructions click here > > < > https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf > >: > > EAI Support queue --> ODP - Enterprise Apps Integration Support > > > > -As a reminder, the Service Availability Managers should be engaged for > > any service impacting issues, with a ***Page*** to > > naitavailabilitym...@officedepot.com or by initiating a MIRT > > > > ------------------------------ > > *From:* Ted Ross <tr...@redhat.com> > > *Sent:* Friday, January 6, 2023 3:37 PM > > *To:* Ekta Awasthi <ekta.awas...@theodpcorp.com> > > *Cc:* users@qpid.apache.org <users@qpid.apache.org>; Ajit Tathawade > > (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale > (Contractor) < > > nilesh.khok...@officedepot.com>; EAIOpsSupport < > > eaiopssupp...@officedepot.com> > > *Subject:* Re: More than configured consumers count > > > > [CAUTION: EXTERNAL SENDER] > > > > Ekta, > > > > One thing you should consider is to bring your Qpid Dispatch Router code > > up to the latest version. You are running 1.16.0 which is pretty old. > > Many bugs have been fixed since 1.16.0. > > > > -Ted > > > > > > On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <ekta.awas...@theodpcorp.com > > > > wrote: > > > > Hello Ted, > > > > Regarding Are the od-broker-1-[ms] connections configured via connectors > > or listeners? ---> They are configured via connectors only below is an > > example. > > > > #od-broker-1-m connector > > connector{ > > name: od-broker-1-m > > host: activemq-test-1.odprivatecloud.com > > port: 61618 > > role: route-container > > linkCapacity: 40 > > sslProfile: od-router-2-test-ssl-profile > > verifyHostName: no #Since we have cert/hostname mismatches > > #saslMechanisms: PLAIN > > } > > > > *Ekta Awasthi*, > > > > ------------------------------ > > *From:* Ted Ross <tr...@redhat.com> > > *Sent:* Tuesday, January 3, 2023 10:43 AM > > *To:* Ekta Awasthi <ekta.awas...@theodpcorp.com> > > *Cc:* users@qpid.apache.org <users@qpid.apache.org>; Ajit Tathawade > > (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale > (Contractor) < > > nilesh.khok...@officedepot.com>; EAIOpsSupport < > > eaiopssupp...@officedepot.com> > > *Subject:* Re: More than configured consumers count > > > > [CAUTION: EXTERNAL SENDER] > > > > Another question... > > > > Are the od-broker-1-[ms] connections configured via connectors or > > listeners? > > > > -Ted > > > > > > On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com> wrote: > > > > Happy New Year Ekta, > > > > Are your micro-services message producers or consumers? > > > > When you see the more-than-expected consumers in Hawtio, get the link > > status from the routers using "qdstat -l". There _should_ be one link > for > > each auto-link as long as the targeted broker is reachable. This should > > provide some clue as to what is happening. > > > > -Ted > > > > > > On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi < > ekta.awas...@theodpcorp.com> > > wrote: > > > > Hello Ted, > > > > Thank you for your response. > > > > We are using AutoLinks for our addresses. Below is an example of a > > autolink queue in qdrouterd.conf for one activemq broker pair. Thanks > > > > address { > > prefix: test-queue > > waypoint: yes > > } > > > > autoLink { > > connection: od-broker-1-m > > addr: test-queue > > dir: in > > } > > > > autoLink { > > connection: od-broker-1-m > > addr: test-queue > > dir: out > > } > > > > autoLink { > > connection: od-broker-1-s > > addr: test-queue > > dir: in > > } > > > > autoLink { > > connection: od-broker-1-s > > addr: test-queue > > dir: out > > } > > > > *Ekta Awasthi*, > > > > ------------------------------ > > *From:* Ted Ross <tr...@redhat.com> > > *Sent:* Tuesday, January 3, 2023 8:47 AM > > *To:* users@qpid.apache.org <users@qpid.apache.org> > > *Cc:* Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh > > Khokale (Contractor) <nilesh.khok...@officedepot.com>; EAIOpsSupport < > > eaiopssupp...@officedepot.com> > > *Subject:* Re: More than configured consumers count > > > > [CAUTION: EXTERNAL SENDER] > > > > Hi Etka, > > > > Can you tell us how you have configured the qdrouters to act as > > consumers? Are you using auto-links or are you using link-routed > addresses? > > > > -Ted > > > > > > On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi > > <ekta.awas...@theodpcorp.com.invalid> wrote: > > > > Hello, > > > > Is this the right mailing DL for QPID related queries. > > > > Just wondering as I have not heard back but given it is a holiday season, > > so I will sit back and wait patiently for someone to reply. > > > > Thanks In Advance. > > Ekta Awasthi > > > > ________________________________ > > From: Ekta Awasthi <ekta.awas...@theodpcorp.com> > > Sent: Thursday, December 22, 2022 1:00 AM > > To: users@qpid.apache.org <users@qpid.apache.org> > > Cc: Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh > > Khokale (Contractor) <nilesh.khok...@officedepot.com>; EAIOpsSupport < > > eaiopssupp...@officedepot.com> > > Subject: More than configured consumers count > > > > Hi There, > > > > We have few issues with the qpid-dispatch router where we are seeing > > multiple consumers are getting created randomly by our qpid dispatch > > routers. We are unable to trace nor able to replicate this issue in our > > lower env's. Below is the architecture diagram. > > > > I apologize if this isn't the right place to ask this question. In case > > there is another place to communicate, do let me know as this is my first > > time reaching out regarding qpid query😀 Please provide any suggestions > or > > feedback to help us understand this issue better. Thanks > > > > Flow: > > Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch > > Routers acting up as consumers in front of brokers (1.16 version) ---> 4 > > Broker pairs of activemq(2.18 version) masters and slaves (independent > > pairs) > > > > Problem statement # 1 > > We are seeing more than the configured number of consumers count in our > > activemq hawtio console causing messages to sit in delivering count which > > are Un browsable since those messages are currently being delivered to > its > > consumers. Having only two qpid dispatch routers(acting as consumers) > > infront of our activemq brokers, the count should always remain 2 but at > > times it is going 3 sometimes 4. To resolve this issue, we are having to > > bounce the qpid’s to release the stuck/bad consumer so that messages can > be > > processed/consumed. > > > > Problem statement # 2 > > At times we see messages going to delivering count even when there are > > only two configured consumers(2qpids) showing in activemq hawtio console. > > We don't know the cause to why the messages get stuck in delivering > count. > > To resolve this, we tried restarting the consumer service but that did > not > > help. Next up we tried restarting the brokers, that did not help and > > noticed that all the stuck delivering messages, the broker slowly > replayed > > back and Eventually the stuck messages came back to delivering count post > > broker restart. To resolve this issue, we are having to bounce the qpid’s > > to release the messages and that fixes the issue. > > > > Version of qpid-dispatch router > > qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm > > qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm > > > > > > Ekta Awasthi > > > > CONFIDENTIALITY NOTICE: The information contained in this email and > > attached document(s) may contain confidential information that is > intended > > only for the addressee(s). If you are not the intended recipient, you are > > hereby advised that any disclosure, copying, distribution or the taking > of > > any action in reliance upon the information is prohibited. If you have > > received this email in error, please immediately notify the sender and > > delete it from your system. > > > > > > CONFIDENTIALITY NOTICE: The information contained in this email and > attached document(s) may contain confidential information that is intended > only for the addressee(s). If you are not the intended recipient, you are > hereby advised that any disclosure, copying, distribution or the taking of > any action in reliance upon the information is prohibited. If you have > received this email in error, please immediately notify the sender and > delete it from your system. >
