shunping opened a new pull request, #40675: URL: https://github.com/apache/beam/pull/40675
Part of #40569. Also noting that there is Bigquery apitools clients are in the process of deprecation #39889. ## Problem `BigQueryWrapper._bigquery_client()` builds the apitools `BigqueryV2` client without a `url`, so it falls back to `BASE_URL = 'https://bigquery.googleapis.com/bigquery/v2/'`. apitools does not read `GOOGLE_CLOUD_UNIVERSE_DOMAIN`, and Beam has no option to override the endpoint. As a result, BigQuery REST calls (tables, jobs, datasets) always go to the public endpoint. This makes BigQuery IO unusable in Trusted Partner Cloud (TPC) and other non-default universes. ## Changes - **New `--bigquery_endpoint` option** in `GoogleCloudOptions`, the Python equivalent of Java's `BigQueryOptions.bigQueryEndpoint` (#32153). It takes the root URL, e.g. `https://bigquery.example.com`. The input is normalized: a missing scheme defaults to `https`, and a trailing `/` or `/bigquery/v2` is stripped. - **Endpoint resolution for `BigqueryV2`**, in this order: 1. `--bigquery_endpoint` 2. `https://bigquery.<domain>` when `GOOGLE_CLOUD_UNIVERSE_DOMAIN` is set to something other than `googleapis.com` 3. The default public URL Step 2 matches how `google-cloud-bigquery` resolves its endpoint, so both clients target the same universe. - **`google.cloud.bigquery.Client`** (streaming inserts) gets `client_options={'api_endpoint': ...}` when `--bigquery_endpoint` is set, merged with `quota_project_id`. When the option is unset, the library resolves the endpoint itself, including `GOOGLE_CLOUD_UNIVERSE_DOMAIN`. - **Quota project header:** the `x-goog-user-project` header from `--quota_project_id` is now also sent to the custom endpoint's host. Previously it was limited to the default BigQuery hosts and would have been silently dropped. - **Worker-side fallback:** when a wrapper is created without pipeline options, e.g. `BigQueryWrapper()` inside DoFns, the endpoint is read from the running pipeline's options via `RuntimeValueProvider`. ## Not changed - The Storage Read API (`BigQueryReadClient`) does not use `--bigquery_endpoint`. It is served from a different host (`bigquerystorage.<domain>`), and the client library already reads `GOOGLE_CLOUD_UNIVERSE_DOMAIN`. Java does apply its option to the storage clients; this can be revisited if a separate storage endpoint option is needed. - Default behavior is unchanged when neither the option nor the env var is set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
