shahar1 commented on code in PR #39203:
URL: https://github.com/apache/beam/pull/39203#discussion_r3828200962
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -929,6 +947,30 @@ def _export_files(self, bq):
return table.schema, metadata_list
+def _create_bq_storage_client(quota_project_id=None):
+ """Create a BigQueryReadClient with optional quota project.
+
+ Args:
+ quota_project_id: Optional GCP project ID to use for quota and billing.
+
+ Returns:
+ A BigQueryReadClient instance.
+ """
+ if quota_project_id:
+ try:
+ import google.auth
+ credentials, _ = google.auth.default()
+ credentials = auth.with_quota_project(credentials, quota_project_id)
+ return bq_storage.BigQueryReadClient(credentials=credentials)
+ except Exception as e:
Review Comment:
Good catch, done in 8c342564919. `auth.with_quota_project` now raises
instead of warning-and-falling-back, and the BigQuery Storage /
google-cloud-bigquery client factories no longer swallow the resulting error.
##########
sdks/python/apache_beam/io/gcp/bigquery_tools.py:
##########
@@ -358,11 +359,17 @@ class BigQueryWrapper(object):
HISTOGRAM_METRIC_LOGGER = MetricLogger()
- def __init__(self, client=None, temp_dataset_id=None, temp_table_ref=None):
- self.client = client or BigQueryWrapper._bigquery_client(PipelineOptions())
- self.gcp_bq_client = client or gcp_bigquery.Client(
- client_info=ClientInfo(
- user_agent="apache-beam-%s" % apache_beam.__version__))
+ def __init__(
+ self,
+ client=None,
Review Comment:
Done in 8c342564919, close to what you sketched: `BigQueryWrapper` now takes
`pipeline_options`/`quota_project_id` and builds both clients itself, with
`client`/`gcp_client` as separate keyword-only test overrides. Added
`_create_bq_wrapper` on both read sources so every wrapper construction goes
through it — this also caught two spots (`estimate_size`, the BEAM_ROW schema
lookup) that were silently dropping the transform-level quota project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
