AlonHarell opened a new issue, #26291: URL: https://github.com/apache/beam/issues/26291
### What would you like to happen? For unbounded streams read from AWS SQS (org.apache.beam.sdk.io.aws2.sqs.SqsIO), messages read from the queue has their visibility timeout extended for a length of about 2 minutes, and then deleted from the queue. Meaning, there's no way to actually signal processing of the message is done or not - for 2 minutes the message will be considered as "in processing", and afterwards the message will be considered done and then deleted from the queue. So for pipelines which require varying processing times, I can't signal wether a message is still processing or done. I'd like to be able to, since currently some messages are automatically deleted from the queue even if they're not done, and some messages aren't deleted even though they're done. Also, side note: Extending messages timeout is done by a blocking loop, so it isn't scalable. ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [X] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [X] Component: IO connector - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
