ahmedabu98 commented on code in PR #24041:
URL: https://github.com/apache/beam/pull/24041#discussion_r1018304092
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQueryHelper.java:
##########
@@ -119,16 +119,21 @@ public static TableReference executeQuery(
}
// Step 2: Create a temporary dataset in the query location only if the
user has not specified
- // a temp dataset.
- String queryJobId =
+ // a temp dataset. The temp table name may be deterministic but the
query job ID has to be
+ // non-deterministic to protect against retries. If BigQuery sees a
repeated query job ID, it
+ // will be skipped.
+ String tempTableID =
BigQueryResourceNaming.createJobIdPrefix(options.getJobName(),
stepUuid, JobType.QUERY);
+ String queryJobId =
+ BigQueryResourceNaming.createJobIdPrefix(
+ options.getJobName(), stepUuid, JobType.QUERY,
BigQueryHelpers.randomUUIDString());
Optional<String> queryTempDatasetOpt =
Optional.ofNullable(queryTempDatasetId);
TableReference queryResultTable =
createTempTableReference(
options.getBigQueryProject() == null
? options.getProject()
: options.getBigQueryProject(),
- queryJobId,
+ tempTableID,
Review Comment:
The table name can stay deterministic, and probably it should because other
parts of BigQuery connector recreate the temp table using just the stepUuid,
e.g. for table deletion [here (export
read)](https://github.com/apache/beam/blob/539fa9159ffc116b2e79e6de2804dfdd1c1e4722/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java#L131-L138)
and [here (direct
read)](https://github.com/apache/beam/blob/539fa9159ffc116b2e79e6de2804dfdd1c1e4722/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1554-L1561).
The problem is when the query job reference is retried with the same name,
so having a non-deterministic name
[here](https://github.com/apache/beam/blob/539fa9159ffc116b2e79e6de2804dfdd1c1e4722/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQueryHelper.java#L166-L173)
is the goal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
