ppawel opened a new issue, #39589: URL: https://github.com/apache/beam/issues/39589
### What happened? Beam 2.75.0 SolaceIO write transform has two "writer" types - batched (sends 50 messages per batch) and streaming (sends one message at a time). With both of those writers, there is the issue when using the writer in a batch-mode Beam pipeline. The issue is that the batch pipeline will frequently end its run before Solace publish results are output by the SolaceIO write transform. In case of `UnboundedBatchedSolaceWriter`, the issue is that it relies on a timer to "finalize" outputing the publish results. In a batch pipeline, timers are not really triggering as expected, at least the pipeline will not wait the 10 seconds which is configured in the code for the timer to fire. Instead, the pipeline will just end - output is lost. In case of `UnboundedStreamingSolaceWriter`, there is no timer but it calls `publishResults` from the `@FinishBundle` method which assumes that by that time there are Solace publish results to output. When there aren't, the batch pipeline again will just end, as there is nothing holding back the watermark and it does not care that there are some Solace ACK callbacks left waiting. Output is lost. CC @stankiewicz @iht ### Issue Priority Priority: 2 (default / most bugs should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [x] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Prism Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [x] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
