Sam Whittle created BEAM-13268:
----------------------------------
Summary: Reduce latency by parallelizing BQ inserts when flushing
due to row limit
Key: BEAM-13268
URL: https://issues.apache.org/jira/browse/BEAM-13268
Project: Beam
Issue Type: Improvement
Components: io-java-gcp
Reporter: Sam Whittle
InsertBatchedElements consumes the output of GroupIntoBatches which flushes
after default 500 elements to respect the max items per streaming insert
request to BQ.
However InsertBatchedElements flushes rows synchrounously meaning that
latencies of writes accumulates. It could instead initiate writes in
ProcessElement and block on write completion in FinishBundle. There could be
some limited configurable parallelism if desired to limit memory usage.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
