github-actions[bot] commented on code in PR #40794:
URL: https://github.com/apache/beam/pull/40794#discussion_r4373279779
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -1948,6 +1955,27 @@ void cleanup(PassThroughThenCleanup.ContextContainer c)
throws Exception {
}
}
}
+
+ /**
+ * Returns the location the extract job ran in, which job lookups
must specify for
+ * regional locations: the source table's dataset location for
table reads, or the query
+ * location for query reads. A query read without an explicit
location returns null, as
+ * before; per {@link TypedRead#withQueryLocation}, that is only
supported for the US
+ * and EU multi-regions, where lookups work without a location.
+ */
+ private @Nullable String getExtractJobLocation(BigQueryOptions
bqOptions)
Review Comment:
**🟠High**: For `BigQueryQuerySourceDef`, the extract job does not only get
a regional location when `withQueryLocation()` is explicitly set. In
`BigQueryQueryHelper.executeQuery`, when `queryLocation == null`, the location
is inferred either from `queryTempDataset` (via
`datasetService.getDataset(projectId, tempDatasetId).getLocation()`) or from
the first referenced table in the query dry-run statistics
(`dryRunJobStats.getQuery().getReferencedTables()`). The temporary dataset is
created with that inferred location, and `BigQuerySourceBase#split`
subsequently looks up that temporary dataset's location to set `bqLocation` on
the extract `JobReference`.
If a user reads from a query against regional tables without calling
`withQueryLocation()` (relying on dry-run table location inference or a
regional `withQueryTempDataset()`), the extract job will be created with the
regional location, but `getExtractJobLocation` here will return `null`, causing
the exact same 404 / 400 cleanup failure.
To ensure the cleanup job lookup always uses the same location as the
extract job, `sourceDef.getTableReference(bqOptions, jobUuid)` (or a helper
method on `BigQuerySourceDef` / `BigQueryQuerySourceDef`) should be used, or
for `BigQueryQuerySourceDef` when `queryLocation == null`, the location should
be resolved the same way (e.g., checking the temp dataset /
`BigQueryQuerySourceDef` cached `tableToExtract` or dry-run tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
