[
https://issues.apache.org/jira/browse/BEAM-10047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niel Markwick updated BEAM-10047:
---------------------------------
Fix Version/s: 2.23.0
Affects Version/s: 2.16.0
2.17.0
2.18.0
2.19.0
2.20.0
2.21.0
2.22.0
Description:
having Spanner IO's grouping, sorting and batching as separate stages makes no
sense as these stages will be fused in any runner that supports fusion.
In addition during these stages, even with fusion, at least 3 full copies of
each mutation are made (serialized then deserialized objects), which leads to
an extremely large use of memory.
Combining these stages would significantly reduce memory usage, and improve
performance.
> SpannerIO: Combine sorting and batching
> ---------------------------------------
>
> Key: BEAM-10047
> URL: https://issues.apache.org/jira/browse/BEAM-10047
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Affects Versions: 2.16.0, 2.17.0, 2.18.0, 2.19.0, 2.20.0, 2.21.0, 2.22.0
> Reporter: Brian Hulette
> Assignee: Niel Markwick
> Priority: P2
> Fix For: 2.23.0
>
>
> having Spanner IO's grouping, sorting and batching as separate stages makes
> no sense as these stages will be fused in any runner that supports fusion.
> In addition during these stages, even with fusion, at least 3 full copies of
> each mutation are made (serialized then deserialized objects), which leads to
> an extremely large use of memory.
>
> Combining these stages would significantly reduce memory usage, and improve
> performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
