scwhittle commented on issue #31373: URL: https://github.com/apache/beam/issues/31373#issuecomment-2137896965
There is a [public gcp issue](https://issuetracker.google.com/321112385) tracking this as well. Beyond using v1, another mitigation allowing the use of .withMethod(STORAGE_WRITE_API) with the v2 runner is to both: - disable autosharding (do not specify .withAutoSharding()) - explicitly specify the write streams, using .withNumStorageWriteApiStreams(n) where n > 0. The value of n must be sufficiently large for desired ingestion rate to BQ but if too large can cause Exceeds 'Concurrent connections' quota errors. You can refer to this [blog-post](https://cloud.google.com/blog/products/data-analytics/streaming-data-into-bigquery-using-storage-write-api) for some guidance on setting the # of streams if you are disabling autosharding. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
