shunping opened a new pull request, #40794:
URL: https://github.com/apache/beam/pull/40794
### Summary
`BigQueryIO.read()` with the default `EXPORT` method runs a cleanup step
after the read that looks up the extract job to find and delete the exported
Avro files. That lookup is built **without a job location**. BigQuery requires
the location to look up jobs in regional locations, so for tables in regional
datasets the lookup fails. Depending on the endpoint, cleanup is either
silently skipped (leaking export files) or fails repeatedly, which stalls
streaming pipelines indefinitely.
This PR sets the job location on that lookup, the same way the export job
itself does.
### What happens today
The export job is created **with** a location
(`BigQuerySourceBase#executeExtract`):
```java
new
JobReference().setProjectId(executingProject).setLocation(bqLocation).setJobId(jobId);
```
But the cleanup in `TypedRead#expand` rebuilds the reference **without** one:
```java
JobReference jobRef = new
JobReference().setProjectId(executingProject).setJobId(...); // no location
Job extractJob =
getBigQueryServices().getJobService(bqOptions).getJob(jobRef);
```
The resulting `jobs.get` has no `location` parameter. For a job in a
regional location:
- **Google Cloud (default universe):** BigQuery returns **404**. `getJob`
treats that as "not found" and returns `null`, so cleanup is **silently
skipped** and the exported files under `<tempLocation>/BigQueryExtractTemp/`
are leaked. This is easy to miss, because datasets in the `US`/`EU`
multi-regions work without a location.
- **Trusted Partner Cloud (TPC):** BigQuery returns **400
`INVALID_ARGUMENT`** instead. `getJob` doesn't treat that as "not found",
retries, and throws. In a streaming pipeline the failed bundle is retried
forever, so the job never makes progress.
Observed on TPC (streaming pipeline reading a table in `u-us-prp1` as a side
input):
```
java.io.IOException: Unable to find BigQuery job:
GenericData{classInfo=[jobId, location, projectId],
{jobId=beam_bq_job_EXPORT_...,
projectId=tpczero-system:dataflow-tpc-test}}, aborting after 9 retries.
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.getJob(BigQueryServicesImpl.java:598)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead$4.cleanup(BigQueryIO.java:1940)
at
org.apache.beam.sdk.io.gcp.bigquery.PassThroughThenCleanup$1.processElement(PassThroughThenCleanup.java:73)
Caused by:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad
Request
GET
https://bigquery.apis-tpczero.goog/bigquery/v2/projects/tpczero-system:dataflow-tpc-test/jobs/beam_bq_job_EXPORT_...
{ "code": 400, "message": "Error: 3848323", "status": "INVALID_ARGUMENT" }
```
### The fix
When building the cleanup's job reference, set the location the export job
ran in:
- **Table reads:** the source table's dataset location
(`BigQueryHelpers.getDatasetLocation`), the same value `BigQuerySourceBase`
uses for the export.
- **Query reads:** the `withQueryLocation` value.
The location is only set when known. Query reads without an explicit
location behave as before. Per the `withQueryLocation` Javadoc, a location must
be specified for queries not executed in `US`/`EU`, and those multi-regions
work without one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
