ahmedabu98 commented on code in PR #28272:
URL: https://github.com/apache/beam/pull/28272#discussion_r1317765971
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java:
##########
@@ -101,27 +99,28 @@ class WriteTables<DestinationT extends @NonNull Object>
@AutoValue
abstract static class Result {
abstract String getTableName();
-
+ // Downstream operations may rely on pane info which will get lost after a
ReShuffle
abstract Boolean isFirstPane();
+
+ abstract Long getPaneIndex();
Review Comment:
Why are we maintaining pane index coming out of write tables? I don't see
that we're using it anywhere downstream
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java:
##########
@@ -206,14 +206,20 @@ private PendingJobData startWriteRename(
// Make sure each destination table gets a unique job id.
String jobIdPrefix =
BigQueryResourceNaming.createJobIdWithDestination(
- c.sideInput(jobIdToken), finalTableDestination, -1);
+ c.sideInput(jobIdToken), finalTableDestination, -1,
c.pane().getIndex());
if (isFirstPane) {
- LOG.info("Setup write disposition {}, create disposition {} for first
pane BigQuery job {}",
- writeDisposition, createDisposition, jobIdPrefix);
+ LOG.info(
+ "Setup write disposition {}, create disposition {} for first pane
BigQuery job {}",
+ writeDisposition,
+ createDisposition,
+ jobIdPrefix);
} else {
- LOG.debug("Setup write disposition {}, create disposition {} for
BigQuery job {}",
- writeDisposition, createDisposition, jobIdPrefix);
+ LOG.debug(
+ "Setup write disposition {}, create disposition {} for BigQuery job
{}",
+ writeDisposition,
+ createDisposition,
+ jobIdPrefix);
Review Comment:
Can we include the bigquery table destination here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
