mosche opened a new pull request #16947: URL: https://github.com/apache/beam/pull/16947
@echauchot @aromanenko-dev Not really a PR (yet), I was just looking into understand usage of schemas vs coders better. The (most) interesting part here is probably `SdkPojoSchema` that works for all the generated AWS models in the SDK v2 (though, in some cases recursive types are used which obviously impose a challenge). I wasn't able to find a lot of good technical documentation (design decisions) for schemas in Beam, though definitely see the value over low level coders! It seems to make sense to bother users as little as possible with coders and [not expose these on any public API](https://docs.google.com/document/d/1V2FkGGunVgvLwi1dKHr-7mtDuwYjTuvuESV_oPzVnfQ/edit?resourcekey=0-KvfQq-5iCcMlu3f3MFJ-GQ#heading=h.xl97lw3dyot7) (M11). Currently coders are all over the place for AWS IOs, with this it seems feasible to drop them (or deprecate). Though, I was wondering a couple of things: - Looks like I cannot register a provider for an interface (`SdkPojo`), that means I have to "announce" it globally using an auto service. - Is there a way a Sink/ WriteIO can set the coder for it's input like that can be done for a Source on the output? Looks like it's only possible to register such a coder globally, regardless if that sink is used or not. - I noticed that a RowCoder cannot be automatically inferred for a row despite every row carrying its schema. That makes sense given runners may use the schema ahead of time for query planning. But it made me wonder if there's any relationship between the schema of the row coder and the runtime schema enforced... If you have any good resources on the topic, I'd be happy to hear ... ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md) GitHub Actions Tests Status (on master branch) ------------------------------------------------------------------------------------------------ [](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule) See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
