Kenneth Jung created BEAM-11208:
-----------------------------------
Summary: BigQuery storage streams fail with QUOTA_EXCEEDED errors
in split
Key: BEAM-11208
URL: https://issues.apache.org/jira/browse/BEAM-11208
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Affects Versions: 2.25.0
Reporter: Kenneth Jung
The BigQueryStorageStreamSource attempts to call
[SplitReadStream](https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1#bigqueryread)
for each splitAtFraction call. However, the storage API
[limits](https://cloud.google.com/bigquery/quotas#storage-limits) the number of
control plane operations per minute for a given project, which can lead to the
service being overloaded in the event of a large pipeline. The stream reader
should not attempt to split a stream once it has learned that the stream can no
longer be split (e.g. once a SplitReadStream call returns an empty response).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
