shahar1 opened a new pull request, #39203: URL: https://github.com/apache/beam/pull/39203
Addresses #37431 Adds support for attributing quota and billing of BigQuery API requests to a specific GCP project (the "quota project", i.e. the `X-Goog-User-Project` header), separate from the project the data resides in. ## Python - New `quota_project_id` parameter on `beam.io.ReadFromBigQuery` (both `EXPORT` and `DIRECT_READ` methods), falling back to a new `--quota_project_id` pipeline option on `GoogleCloudOptions`. - Credentials are wrapped via `google.auth` `with_quota_project()` (new helper `apache_beam.internal.gcp.auth.with_quota_project`); applies to the apitools BigQuery client, the `google-cloud-bigquery` client, and the BigQuery Storage read client. ## Java - New `--bigQueryQuotaProjectId` option on `BigQueryOptions` (named after the existing `--bigQueryProject`, which controls the *job* parent project — a related but distinct concept). - Applied in `BigQueryServicesImpl`: the HTTP BigQuery client wraps its credentials with `GoogleCredentials.createWithQuotaProject(...)`, and the Storage read/write gRPC clients set `quotaProjectId` on their gax settings. Because the HTTP job/dataset services are shared between read and write paths, the option uniformly covers all BigQuery API calls made by the SDK. - Per-transform configuration (e.g. `TypedRead.withQuotaProjectId(...)`) would require threading the value through the `BigQueryServices` interface and its fakes; left as a follow-up. ## Go - New `bigqueryio.WithQuotaProject(...)` query option, accepted by both `bigqueryio.Read` (signature extended with variadic options — backward compatible) and `bigqueryio.Query`, applied to the client via `option.WithQuotaProject`. ## TypeScript Not covered here: the TypeScript BigQuery IO delegates to the Java `schemaio_bigquery_read:v1` expansion transform, so per-transform support there depends on the Java per-transform follow-up plus a `BigQuerySchemaIOProvider` configuration-schema change. ## Testing - Python: `bigquery_test.py`, `bigquery_tools_test.py`, `auth_test.py` pass locally (25 quota-specific tests). - Java: `BigQueryServicesImplTest` passes locally, including new tests asserting the quota project on Storage client settings and on wrapped credentials; checkstyle/spotless clean. - Go: unit tests and `go vet` pass. ## Generative AI disclosure Parts of this PR (the Java and Go implementations, rebase conflict resolution, and test/lint fixes) were developed with the assistance of Claude Code (Anthropic, model `claude-fable-5`), per the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html). The corresponding commits carry `Generated-by:` trailers. All changes were reviewed, and tested locally, by the author. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead. - [x] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
