Eliaaazzz opened a new pull request, #40154:
URL: https://github.com/apache/beam/pull/40154
Part of #39684. Depends on #40153 (the first two commits); review the last
commit.
The Go SDK could only support a portable logical type by registering it in
three places (`schema.RegisterLogicalType`, `coder.RegisterSchemaProviders`,
and the `beam.SchemaProvider` glue) with hand written encoders, and none of the
standard logical types from schema.proto were registered. A schema from another
SDK with a `beam:logical_type:date:v1` or `beam:logical_type:micros_instant:v1`
field failed with `unknown logical type`.
Registration API, in package `schema`:
- `RegisterLogicalTypeConversion[GoT, StorageT](urn, toStorage,
fromStorage)` registers the URN, the Go type and the storage type, and builds
the row coder for `GoT` from the row field coder of `StorageT` plus the two
conversion functions.
- `RegisterParameterizedLogicalTypeConversion` is the same for a logical
type with an argument; each argument value gets its own Go type.
- `RegisterLogicalTypeCoder[GoT, StorageT](urn, enc, dec)` is for logical
types whose wire encoding differs from the row encoding of the storage type,
such as `millis_instant` and `decimal` (following PRs).
- `coder.RowFieldEncoderForType` and `coder.RowFieldDecoderForType` expose
the row field encoding of a single type, which the registration builds on.
Standard types, registered by default:
- `schema.Date` (`beam:logical_type:date:v1`), a calendar date stored as
days since the epoch.
- `schema.MicrosInstant` (`beam:logical_type:micros_instant:v1`), a
`time.Time` stored as a row of seconds and microseconds. Encoding a value with
sub microsecond precision returns an error, as in the Java SDK.
`time.Time` keeps its current Go specific encoding; the standard types are
separate Go types, so existing pipelines are unaffected. Because the Go row
coder is built from the Go type alone, each URN maps to one named Go type.
Regression test changes in `sdks/go/test/regression/coders/fromyaml`:
- The expected row of a `beam:coder:row:v1` case is built from the schema in
the coder payload through `schema.ToType`, so logical type fields get their
registered Go types. This replaces the name based type table.
- Coder payloads are decoded from the yaml string the same way the examples
are, so payload bytes above 0x7f (used by the `timestamp:v1` cases) are read
correctly.
- Encoding mismatches now count as failures; before, they were only logged.
- The `micros_instant` case runs. The remaining logical type cases are
skipped by URN with a reason pointing at #39684.
Tests: schema conversion and byte level row encoding for `Date` and
`MicrosInstant` (the bytes are the `micros_instant` example of
standard_coders.yaml), date conversion at the epoch boundaries, the sub
microsecond error, `RegisterLogicalTypeConversion` end to end with a user type,
and `RowFieldEncoderForType`/`RowFieldDecoderForType` for atomic, slice and
struct types.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
