ppawel commented on issue #31440: URL: https://github.com/apache/beam/issues/31440#issuecomment-2200236386
@bzablocki Very cool to see Solace related development in Beam! We are using Solace and Beam together for ~3 years now and we've been on a journey... the connector made by Solace (https://github.com/SolaceProducts/solace-apache-beam) is not maintained anymore, so we switched to JMS IO but that has its own issues... so we have to patch JmsIO for our use cases. If you have time, you could take a look at my comments here, might be useful to have more view on the use cases: https://github.com/apache/beam/pull/30218#issuecomment-2037429676 In general, the pain points with JmsIO are: 1. How it manages JMS resources - since Beam 2.55, it closes the JMS session every time it acknowledges a message (which means every ~10 messages based on what I see in local and GCP runners). 2. How it manages the watermark - I had to patch it so it behaves more like PubsubIO, main problem is that if no messages are coming, it never bumps the watermark by itself so then windowing behaves very strangely (i.e. windows don't close) in pipelines in streaming mode. 3. The integration tests I have seen there assume that the JMS client does not use buffering (a.k.a. prefetch, or "window size" in Solace terms) and I think that's fundamentally wrong. Solace client does not even allow to switch off buffering and you can see in my comment linked above how it turns out (deadlock since Beam 2.55). Anyway, those are just some random thoughts, hopefully useful :) For now I am maintaining my custom patches and hope to one day upstream them (no time until now). Hopefully when SolaceIO is ready, we can switch to it and all our problems will be magically solved ;) #nopressure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
