goutamadwant opened a new pull request, #40119:
URL: https://github.com/apache/beam/pull/40119
Fixes #40110.
Storage Write API failed rows currently contain TIMESTAMP strings with both
a space and a `T`, but no time zone. Converting those rows to Beam Rows throws
instead of delivering the rejected records.
Print standard TIMESTAMP values in BigQuery's canonical UTC format, with at
most six fractional digits. Accept that format in the `SqlTypes.TIMESTAMP`
consumer while retaining its ISO and integer-microsecond inputs. Protobuf
Timestamp values are truncated to microsecond precision before printing.
DATETIME's numeric-seconds behavior and TimestampPicos output are unchanged.
Replace the malformed writer-test expectations and the fixture normalizer's
use of the production formatter with literal expected values.
## Testing
The new `StorageApiTimestampWriteTest` runs public BigQueryIO pipelines with
a fake append service in batch, fixed-shard streaming, and at-least-once modes.
It rejects two timestamp-bearing records on successive appends, then accepts
the remaining record. Assertions cover failed rows, error messages, successful
output, persisted rows, and shrinking append batches. Failed rows are decoded
through both `SqlTypes.TIMESTAMP` and DATETIME consumers.
All three modes fail with `DateTimeParseException` on unchanged production
code and pass with this fix. The two conversion test classes and the pipeline
regression pass together: 93 tests. They also cover integer/protobuf
timestamps, sub-microsecond input, pre-epoch and BigQuery range boundaries,
nested/repeated/null fields, input compatibility, and TimestampPicos
preservation.
The four-class rerun below passed all 33 applicable executions of the
modified writer methods, including Avro, protobuf, and failed-row cases.
Overall: 391 passed, 231 skipped by existing assumptions, and one failure in
`testAutoPatchTableSchemaTest[3]`. That test also fails in isolation on the
pristine base, but with a different symptom: missing nested output fields
rather than a missing-required-field retry error. The identical isolated run on
patched code passes both applicable cases. These outcomes vary by run/context;
the cause remains unresolved, so broader validation is not fully green.
Spotless and changelog validation passed.
```sh
./gradlew :sdks:java:io:google-cloud-platform:test \
--tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest \
--tests org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProtoTest \
--tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtilsTest \
--tests org.apache.beam.sdk.io.gcp.bigquery.StorageApiTimestampWriteTest \
:sdks:java:io:google-cloud-platform:spotlessJavaCheck validateChanges \
--continue --no-build-cache --no-parallel --max-workers=1
```
No live BigQuery or Dataflow validation was run.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description.
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md).
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
