kumarrk21 opened a new issue, #22279:
URL: https://github.com/apache/beam/issues/22279
### What happened?
When writing data to BigQuery using BigQueryIO storage API, no errors are
returned to WriteResult even if rows are not written to BigQuery. This happens
during schema mismatch between incoming data and BQ Schema. Steps to reproduce
is provided below
1) Create a BQ table with just one string field in it (e.g. field1)
2) Send a JSON payload to PubSub as below and write a pipeline to pull this
PubSub message and write to BQ [Create TableRow dynamically by parsing the JSON
payload]
```
{
"field1" : "1"
}
```
3) Verify if records were successfully create in BQ. It should.
4) Now send another JSON payload to PubSub but with an additional field
```
{
"field1" : "1",
"field2" : "2"
}
```
5) No records will be inserted in BQ because there is a schema mismatch
between incoming data and BQ schema
Expected result: BigQueryIO should return errors in WriteResult
Observed result: No errors are returned in WriteResult. Also, no records are
written to BigQuery
Provide below is the code snippet
```
WriteResult writeResult = tableRows.apply(
BigQueryIO.writeTableRows()
.to(TABLESPEC)
.withSchema(BigQueryUtils.toTableSchema(BQSchema))
.optimizedWrites()
.withExtendedErrorInfo()
.withMethod(Method.STORAGE_API_AT_LEAST_ONCE)
.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
.withWriteDisposition(WriteDisposition.WRITE_APPEND));
```
### Issue Priority
Priority: 1
### Issue Component
Component: io-java-gcp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
