Thanks for putting this together, it will be a really useful feature to
have.
I am in favor of the string-pattern approaches. I think we need to support
both the {record=..., dest_info=...} and the elide-fields approaches, as
the former is nicer when one has a fixed representation for the
output record (e.g. a proto or avro schema) and the flattened form for ease
of use in more free-form contexts (e.g. when producing records from YAML
and SQL).
Also left some comments on the doc.
On Wed, Mar 27, 2024 at 6:51 AM Ahmed Abualsaud via dev <dev@beam.apache.org>
wrote:
> Hey all,
>
> There have been some conversations lately about how best to enable dynamic
> destinations in a portable context. Usually, this comes up for
> cross-language transforms and more recently for Beam YAML.
>
> I've started a short doc outlining some routes we could take. The purpose
> is to establish a good standard for supporting dynamic destinations with
> portability, one that can be applied to most use cases and IOs. Please take
> a look and add any thoughts!
>
> https://s.apache.org/portable-dynamic-destinations
>
> Best,
> Ahmed
>
