gabihodoroaga opened a new issue, #23028:
URL: https://github.com/apache/beam/issues/23028
### What happened?
I encountered this error when I updated my BigQuery batch pipeline write
method from `FILE_LOADS` to `STORAGE_WRITE_API`
```text
Exception in thread "main"
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.RuntimeException:
java.lang.RuntimeException: java.lang.RuntimeException:
com.google.api.gax.rpc.PermissionDeniedException:
io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission denied:
Consumer 'project:null' has been suspended.
```
After a while I realized that the error is generated by the fact that I’m
not setting the project name when I set the destination. For example
`dataset.table` instead of `project-id:dataset.table`.
This breaks the compatibility with the current pipelines and it will make it
harder for people to make the transition to the new BigQuery Storage Write API.
By digging into the source code I found the this situation is handled by the
FILE_LOADS by using the project property from the pipeline options
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java:239
```java
if (Strings.isNullOrEmpty(tableReference.getProjectId())) {
BigQueryOptions options =
c.getPipelineOptions().as(BigQueryOptions.class);
tableReference.setProjectId(
options.getBigQueryProject() == null
? options.getProject()
: options.getBigQueryProject());
tableDestination =
tableDestination.withTableReference(tableReference);
}
```
For the Storage API implementation this can be handled by updating this
function and pass the pipeline options as parameter
```java
public String getTableUrn() {
TableReference table = getTableReference();
return String.format(
"projects/%s/datasets/%s/tables/%s",
table.getProjectId(), table.getDatasetId(), table.getTableId());
}
```
I can try to make a PR if this approach is acceptable.
### Issue Priority
Priority: 3
### Issue Component
Component: io-java-gcp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
