Abacn commented on issue #30646: URL: https://github.com/apache/beam/issues/30646#issuecomment-2000410578
This is a known issue and there was another approach trying to address it before, namely #24260. The current status of this approach is a pipeline option `--enableStorageReadApiV2` to enroll in BigQuery storage Read API v2, where the read streams are the units of work instead of unit of parallelism, and the streams that created once won't be split further. This appraoch still need stress test to see if it mitigated the issue and to what extent. This task is try to resolve the issue by client side throttling, alternative to aforementioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
