Vaishnavi-Pandey12 commented on code in PR #37108: URL: https://github.com/apache/beam/pull/37108#discussion_r2618968148
########## website/www/site/content/en/documentation/io/built-in/google-bigquery.md: ########## @@ -659,6 +659,11 @@ runtime. The sharding behavior depends on the runners. You must use `triggering_frequency` to specify a triggering frequency for initiating load jobs. Be careful about setting the frequency such that your pipeline doesn't exceed the BigQuery load job [quota limit](https://cloud.google.com/bigquery/quotas#load_jobs). + +> **Note:** When using file load-based BigQuery writes with dynamic destinations and a non-zero +> `triggering_frequency`, temporary tables may be created repeatedly and loads +> are not finalized into destination tables. This is a known limitation (see BEAM-9917). Review Comment: Thanks for the clarification — that makes sense. I added the documentation note to clarify the current behavior while investigating the issue, but I understand that the primary goal is to address the root cause. As a next step, I can add a failing unit test that captures the current behavior with dynamic destinations and triggering_frequency, or help with investigation into the finalization logic. Please let me know which direction would be preferred. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
