[
https://issues.apache.org/jira/browse/BEAM-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111506#comment-17111506
]
Brian Hulette commented on BEAM-9822:
-------------------------------------
Spoke with Niel about this offline. This actually does represent a
(long-standing) regression where streaming writes to SpannerIO have high
latency and tend to OOM after a change in Beam 2.4.
[~nielm] I feel ok merging #11529 and #11532 since they're relatively small
diffs and I have a handle on them. But #11570 is much bigger and I think it's
unlikely we'll get it reviewed today or tomorrow. Is it ok if that one doesn't
make it into 2.22? It seems the first two should be sufficient to unblock
streaming spanner writes.
If you are ok with waiting on #11570, I think we should pull out the doc
changes from it into a separate PR that we can merge tomorrow before the
release cut (and this could include an update to CHANGES.md as well). How does
that sound?
> SpannerIO: Reduce memory usage - especially when streaming
> ----------------------------------------------------------
>
> Key: BEAM-9822
> URL: https://issues.apache.org/jira/browse/BEAM-9822
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.20.0, 2.21.0
> Reporter: Niel Markwick
> Assignee: Niel Markwick
> Priority: P2
> Labels: google-cloud-spanner
> Fix For: 2.22.0
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> SpannerIO uses a lot of memory.
> In Streaming Dataflow, it uses many times as much (because dataflow creates
> many worker threads)
> Lower the memory use, and change default parameters during streaming to use
> smaller batches and disable grouping.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
