[
https://issues.apache.org/jira/browse/BEAM-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Moritz Mack updated BEAM-13824:
-------------------------------
Description:
In BEAM-13203 {{SnsIO.writeAsync}} was deprecated to address the risk of data
loss in that transform. Unfortunately, it has considerable issues in design and
implementation with respect to the beam programming model as well as general
async programming:
* Output for a bundle may be emitted even after the bundle is finished (lack
of join / waiting for pending futures), additionally output is emitted from
multiple threads risking concurrent modification issues on underlying data
structures.
* Exceptions disappear into the void (respective tests are implemented
incorrectly as they don't throw asynchronously and could make one think this
works).
* Parallelism is unlimited causing the writer to instantly accumulate
everything into memory.
With BEAM-13563 it will be possible to change {{SnsIO.write}} into a more
performant async implementation without any breaking changes to the public user
facing API making {{SnsIO.writeAsync}} obsolete.
> Migrate SnsIO.write to async implementation
> -------------------------------------------
>
> Key: BEAM-13824
> URL: https://issues.apache.org/jira/browse/BEAM-13824
> Project: Beam
> Issue Type: Improvement
> Components: io-java-aws
> Reporter: Moritz Mack
> Priority: P2
> Labels: aws-sdk-v2
>
> In BEAM-13203 {{SnsIO.writeAsync}} was deprecated to address the risk of
> data loss in that transform. Unfortunately, it has considerable issues in
> design and implementation with respect to the beam programming model as well
> as general async programming:
> * Output for a bundle may be emitted even after the bundle is finished (lack
> of join / waiting for pending futures), additionally output is emitted from
> multiple threads risking concurrent modification issues on underlying data
> structures.
> * Exceptions disappear into the void (respective tests are implemented
> incorrectly as they don't throw asynchronously and could make one think this
> works).
> * Parallelism is unlimited causing the writer to instantly accumulate
> everything into memory.
> With BEAM-13563 it will be possible to change {{SnsIO.write}} into a more
> performant async implementation without any breaking changes to the public
> user facing API making {{SnsIO.writeAsync}} obsolete.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
