damccorm opened a new issue, #21366:
URL: https://github.com/apache/beam/issues/21366
In BEAM-13203 `SnsIO.writeAsync` was deprecated to address the risk of data
loss in that transform. Unfortunately, it has considerable issues in design and
implementation with respect to the beam programming model as well as general
async programming:
* Output for a bundle may be emitted even after the bundle is finished
(lack of join / waiting for pending futures), additionally output is emitted
from multiple threads risking concurrent modification issues on underlying data
structures.
* Exceptions disappear into the void (respective tests are implemented
incorrectly as they don't throw asynchronously and could make one think this
works).
* Parallelism is unlimited causing the writer to instantly accumulate
everything into memory.
With BEAM-13563 it will be possible to change `SnsIO.write` into a more
performant async implementation without any breaking changes to the public user
facing API making `SnsIO.writeAsync` obsolete.
Imported from Jira
[BEAM-13824](https://issues.apache.org/jira/browse/BEAM-13824). Original Jira
may contain additional context.
Reported by: mosche.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
