Ah this is super useful context, thank you! I can submit a couple PRs to get AvroIO.sink up to parity if that's the way forward.
On Tue, Sep 13, 2022 at 2:53 PM John Casey via user <user@beam.apache.org> wrote: > Hi Steve, > > I've asked around, and it looks like this confusing state is due to a > migration that isn't complete (and likely won't be until Beam 3.0). > > Here is the doc that explains some of the history: > https://docs.google.com/document/d/1zcF4ZGtq8pxzLZxgD_JMWAouSszIf9LnFANWHKBsZlg/edit > And a PR that implements some of the changes: > https://github.com/apache/beam/pull/3817 > > Based on this, AvroIO.sink is what we recommend. Please feel free to raise > issues on Github to account for features you're missing. In addition, if > you think they are straightforward changes, I'd be happy to discuss > designs, or look at proposed changes to make these features available. > > I hope this helps, > John > > On Mon, Sep 12, 2022 at 3:38 PM Steve Niemitz <sniem...@apache.org> wrote: > >> We're trying to do some semi-advanced custom logic (custom writers and >> schemas per destination) with AvroIO, and want to use >> DynamicAvroDestinations to accomplish this. >> >> However, AvroIO.to(DynamicAvroDestinations) is deprecated, but there >> doesn't seem to be any other way to accomplish what we want here. >> AvroIO.sink is much less sophisticated than the non-sink options, missing >> much of the configurability that the non-sink version has. For example, >> there's no way to project from the UserT -> OutputT with the sink version, >> only from UserT -> GenericRecord, which isn't what we want. >> >> It seems like most things would be trivial to fix or add on the >> AvroIO.sink implementation, is that the intended way that people would be >> consuming AvroIO? I'm a little confused with FileIO.write/writeDynamic vs >> WriteFiles vs AvroIO.write, some seem deprecated, and some seem >> not-deprecated-but-not-recommended. To add to the confusion AvroIO.write >> uses WriteFiles, but the documentation for the deprecated >> AvroIO.to(DynamicAvroDestinations) points to FileIO.write. Which is the >> "right" one to use? >> >
