Abacn commented on code in PR #31823:
URL: https://github.com/apache/beam/pull/31823#discussion_r1672668041
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:
##########
@@ -1089,6 +1098,8 @@ public void process(
initializedDatasetService,
initializedWriteStreamService,
pipelineOptions.as(BigQueryOptions.class)));
+ Lineage.getSinks()
Review Comment:
TableDestination can have null projectId that was why
"tableDestination1.getTableUrn(bigQueryOptions)" was passed in here, which has
a couple of fallback logics to resolve the full destination.
A potential issue for the current change if the sink is in a different
project set by pipeline option, the metrics could report to a wrong destination
However I also notice that Lineage is per project basis. How to handle cross
project observability may be an open question.
If you think this is fine, the change is good for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
