nguymin4 opened a new pull request, #34761: URL: https://github.com/apache/beam/pull/34761
Support configuring flush_count and max_row_bytes of WriteToBigTable ------------------------ Related to this issue: https://github.com/apache/beam/issues/34760 Our pipeline write 1.3 millions data points every 5 mins with a short burst in < 1 min. Since upgrading to 2.64.0 we started observing this error `google.api_core.exceptions.ResourceExhausted: 429 You have reached the limit of total mutations in your queue. Throttle your usage and wait for operations to finish` I guess with current FLUSH_COUNT=1000 it's too low for our usage and we want to be able to configure this `flush_count` option which will probably solve the `429` error above. GitHub Actions Tests Status (on master branch) ------------------------------------------------------------------------------------------------ [](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule) See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
