I have also created a qpid ticket on QPID JIRA Board. https://issues.apache.org/jira/browse/DISPATCH-2353
Ekta Awasthi, Engineer, EAI Operations & Support | Office Depot, Inc. 6600 North Military Trail | Boca Raton, FL 33496-2434 Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awas...@officedepot.com<mailto:ekta.awas...@officedepot.com> -- Tips for EAI Support Engagement -- -EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS> -EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support -As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to naitavailabilitym...@officedepot.com<mailto:naitavailabilitym...@officedepot.com> or by initiating a MIRT ________________________________ From: Ekta Awasthi <ekta.awas...@theodpcorp.com> Sent: Wednesday, January 11, 2023 1:35 PM To: Ted Ross <tr...@redhat.com> Cc: users@qpid.apache.org <users@qpid.apache.org>; Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale (Contractor) <nilesh.khok...@officedepot.com>; EAIOpsSupport <eaiopssupp...@officedepot.com> Subject: Re: More than configured consumers count Hello Ted, I agree with you, we can certainly try upgrading but since this issue is happening directly in PROD and so frequently, we need a quick solution to solve these issues. Any help/suggestions will be greatly appreciated. Ekta Awasthi, Engineer, EAI Operations & Support | Office Depot, Inc. 6600 North Military Trail | Boca Raton, FL 33496-2434 Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awas...@officedepot.com<mailto:ekta.awas...@officedepot.com> -- Tips for EAI Support Engagement -- -EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS> -EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support -As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to naitavailabilitym...@officedepot.com<mailto:naitavailabilitym...@officedepot.com> or by initiating a MIRT ________________________________ From: Ted Ross <tr...@redhat.com> Sent: Friday, January 6, 2023 3:37 PM To: Ekta Awasthi <ekta.awas...@theodpcorp.com> Cc: users@qpid.apache.org <users@qpid.apache.org>; Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale (Contractor) <nilesh.khok...@officedepot.com>; EAIOpsSupport <eaiopssupp...@officedepot.com> Subject: Re: More than configured consumers count [CAUTION: EXTERNAL SENDER] Ekta, One thing you should consider is to bring your Qpid Dispatch Router code up to the latest version. You are running 1.16.0 which is pretty old. Many bugs have been fixed since 1.16.0. -Ted On Tue, Jan 3, 2023 at 5:03 PM Ekta Awasthi <ekta.awas...@theodpcorp.com<mailto:ekta.awas...@theodpcorp.com>> wrote: Hello Ted, Regarding Are the od-broker-1-[ms] connections configured via connectors or listeners? ---> They are configured via connectors only below is an example. #od-broker-1-m connector connector{ name: od-broker-1-m host: activemq-test-1.odprivatecloud.com<http://activemq-test-1.odprivatecloud.com> port: 61618 role: route-container linkCapacity: 40 sslProfile: od-router-2-test-ssl-profile verifyHostName: no #Since we have cert/hostname mismatches #saslMechanisms: PLAIN } Ekta Awasthi, ________________________________ From: Ted Ross <tr...@redhat.com<mailto:tr...@redhat.com>> Sent: Tuesday, January 3, 2023 10:43 AM To: Ekta Awasthi <ekta.awas...@theodpcorp.com<mailto:ekta.awas...@theodpcorp.com>> Cc: users@qpid.apache.org<mailto:users@qpid.apache.org> <users@qpid.apache.org<mailto:users@qpid.apache.org>>; Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com<mailto:ajit.tathaw...@officedepot.com>>; Nilesh Khokale (Contractor) <nilesh.khok...@officedepot.com<mailto:nilesh.khok...@officedepot.com>>; EAIOpsSupport <eaiopssupp...@officedepot.com<mailto:eaiopssupp...@officedepot.com>> Subject: Re: More than configured consumers count [CAUTION: EXTERNAL SENDER] Another question... Are the od-broker-1-[ms] connections configured via connectors or listeners? -Ted On Tue, Jan 3, 2023 at 10:14 AM Ted Ross <tr...@redhat.com<mailto:tr...@redhat.com>> wrote: Happy New Year Ekta, Are your micro-services message producers or consumers? When you see the more-than-expected consumers in Hawtio, get the link status from the routers using "qdstat -l". There _should_ be one link for each auto-link as long as the targeted broker is reachable. This should provide some clue as to what is happening. -Ted On Tue, Jan 3, 2023 at 10:01 AM Ekta Awasthi <ekta.awas...@theodpcorp.com<mailto:ekta.awas...@theodpcorp.com>> wrote: Hello Ted, Thank you for your response. We are using AutoLinks for our addresses. Below is an example of a autolink queue in qdrouterd.conf for one activemq broker pair. Thanks address { prefix: test-queue waypoint: yes } autoLink { connection: od-broker-1-m addr: test-queue dir: in } autoLink { connection: od-broker-1-m addr: test-queue dir: out } autoLink { connection: od-broker-1-s addr: test-queue dir: in } autoLink { connection: od-broker-1-s addr: test-queue dir: out } Ekta Awasthi, ________________________________ From: Ted Ross <tr...@redhat.com<mailto:tr...@redhat.com>> Sent: Tuesday, January 3, 2023 8:47 AM To: users@qpid.apache.org<mailto:users@qpid.apache.org> <users@qpid.apache.org<mailto:users@qpid.apache.org>> Cc: Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com<mailto:ajit.tathaw...@officedepot.com>>; Nilesh Khokale (Contractor) <nilesh.khok...@officedepot.com<mailto:nilesh.khok...@officedepot.com>>; EAIOpsSupport <eaiopssupp...@officedepot.com<mailto:eaiopssupp...@officedepot.com>> Subject: Re: More than configured consumers count [CAUTION: EXTERNAL SENDER] Hi Etka, Can you tell us how you have configured the qdrouters to act as consumers? Are you using auto-links or are you using link-routed addresses? -Ted On Wed, Dec 28, 2022 at 11:20 AM Ekta Awasthi <ekta.awas...@theodpcorp.com.invalid> wrote: Hello, Is this the right mailing DL for QPID related queries. Just wondering as I have not heard back but given it is a holiday season, so I will sit back and wait patiently for someone to reply. Thanks In Advance. Ekta Awasthi ________________________________ From: Ekta Awasthi <ekta.awas...@theodpcorp.com<mailto:ekta.awas...@theodpcorp.com>> Sent: Thursday, December 22, 2022 1:00 AM To: users@qpid.apache.org<mailto:users@qpid.apache.org> <users@qpid.apache.org<mailto:users@qpid.apache.org>> Cc: Ajit Tathawade (CompuCom) <ajit.tathaw...@officedepot.com>; Nilesh Khokale (Contractor) <nilesh.khok...@officedepot.com>; EAIOpsSupport <eaiopssupp...@officedepot.com<mailto:eaiopssupp...@officedepot.com>> Subject: More than configured consumers count Hi There, We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram. I apologize if this isn't the right place to ask this question. In case there is another place to communicate, do let me know as this is my first time reaching out regarding qpid query😀 Please provide any suggestions or feedback to help us understand this issue better. Thanks Flow: Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs) Problem statement # 1 We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed. Problem statement # 2 At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue. Version of qpid-dispatch router qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm Ekta Awasthi CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.
