Any chance to share how you set up auto scaler and how many splits you are setting for this pipeline?
https://github.com/apache/beam/blob/master/runners%2Fgoogle-cloud-dataflow-java%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Frunners%2Fdataflow%2Finternal%2FCustomSources.java#L95 here we use dataflow pipeline options to specify amount of splits. wt., 28 sty 2025, 18:47 użytkownik Zack Culberson < zack.culber...@albertsons.com> napisał: > Just to add this as well > > > > So our middle queue has 7122 messages sitting on it but either the JMS IO > didn’t scale up to handle those or our first queue that has 734 started > getting more messages either way I was expecting all the Queue managers to > behave the like the first to keep up with the throughput but that’s not > what Im observing. > > Thanks > Zack Culberson > > > > *From:* Radek Stankiewicz <radosl...@google.com> > *Sent:* Tuesday, January 28, 2025 9:15 AM > *To:* Ahmet Altay <al...@google.com> > *Cc:* user@beam.apache.org; Yi Hu <ya...@google.com>; Zack Culberson < > zack.culber...@albertsons.com> > *Subject:* EXTERNAL Email: Re: JMSIO support > > > > hey Zack, > > JMSIO reader reads messages in a loop and it looks like whenever there is > a checkpoint Beam recreates the connection. Maybe it takes some time for > the driver and MQ to finalize the connection. > > Can you confirm which dataflow runner and sdk you are using? > > When you observed 2000 connections, do I understand correctly that this QM > had empty queues? > > Were 2000 connections reported by QM monitoring? > > > > Radek > > > > > > > > On Tue, Jan 28, 2025 at 1:28 AM Ahmet Altay <al...@google.com> wrote: > > I do not know the answer. Adding @Radek Stankiewicz <radosl...@google.com> > & @Yi Hu <ya...@google.com> in case they might be able to help. > > > > On Mon, Jan 27, 2025 at 2:55 PM Zack Culberson < > zack.culber...@albertsons.com> wrote: > > Hi all, > > I was wondering if anyone would know, we are using the JMS IO read to read > from 3 IBM MQ’s. We setup the job to have a configurable amount of readers, > which right now is 9 so three readers per mq. Our question came that at one > point we were seeing the number of connections for one of our Queue > managers was going up to 2000 connections but all of the messages were > going to another Queue Manager whose connections stayed at about a constant > 100. Could some one help me understand why that is and does the JMS leave > the connection open when it connects to read or does it open and close > every time it connects ? > > Thank you > > > > *Zack Culberson* > > > ------------------------------ > > Warning: All e-mail sent to this address will be received by the corporate > e-mail system, and is subject to archival and review by someone other than > the recipient. This e-mail may contain proprietary information and is > intended only for the use of the intended recipient(s). If the reader of > this message is not the intended recipient(s), you are notified that you > have received this message in error and that any review, dissemination, > distribution or copying of this message is strictly prohibited. If you have > received this message in error, please notify the sender immediately. > ------------------------------ > > ------------------------------ > Warning: All e-mail sent to this address will be received by the corporate > e-mail system, and is subject to archival and review by someone other than > the recipient. This e-mail may contain proprietary information and is > intended only for the use of the intended recipient(s). If the reader of > this message is not the intended recipient(s), you are notified that you > have received this message in error and that any review, dissemination, > distribution or copying of this message is strictly prohibited. If you have > received this message in error, please notify the sender immediately. > ------------------------------ >
