Adding Zack explicitly. On Fri, Nov 15, 2024 at 2:57 AM Radek Stankiewicz via user < user@beam.apache.org> wrote:
> hey Zach, > > Few recommendations: > - level of parallelism dataflow v1 is set > by desiredNumUnboundedSourceSplits (default N=0) which means that JmsIO > will create N splits equal to the amount of your workers. Feel free to > increase it. > - you should implement AutoScaler > <https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/AutoScaler.html> > and set it via withAutoScaler - what JMS backend you are using? Some > backends like MQ provide queue length so to estimate backlog you need to > calculate moving average of payload size. > - if you are using Solace, there is a new SolaceIO created. > > 2 msg/s is suspiciously low - have you tried to debug the pipeline with > profiling to understand where the pipeline is wasting time? > > Radek > > On 2024/11/14 01:51:31 Zack Culberson wrote: > > Hi All, > > > > I did end up getting my issue resolved with the kafka writer so thank > you for that, I have some questions on JMS io as I am trying to increase > the rate we are reading messages from the Queue we have. Right now it tops > out at about 2msg/s and I am running on a n2-standard-8 machine. I have > tried increasing the MaxBatchSize and that hasn't seemed to help. I was > wondering on how to setup and use an autoscaler and is that the best course > of action if we are looking to be reading 500 msg/s at peak time ? Any help > on what I could do here to improve throughput I would create appreciate. > > > > > > Zack Culberson > > Data Engineer, Retail > > zackery.culber...@albertsons.com<ma...@albertsons.com> > > Albertsons Companies > > Phone: +1.605.431.1112 <(605)%20431-1112> > > [image005] > > > > ________________________________ > > Warning: All e-mail sent to this address will be received by the > corporate e-mail system, and is subject to archival and review by someone > other than the recipient. This e-mail may contain proprietary information > and is intended only for the use of the intended recipient(s). If the > reader of this message is not the intended recipient(s), you are notified > that you have received this message in error and that any review, > dissemination, distribution or copying of this message is strictly > prohibited. If you have received this message in error, please notify the > sender immediately. > > ________________________________ > > >
