I would appreciate if someone would look at the following PR and get it to master:
https://github.com/apache/beam/pull/10413# a lot of work needs to follow, but if we have the base already on master the next layers can follow. As a reminder, this is the base proposal: https://docs.google.com/document/d/1yCCRU5pViVQIO8-YAb66VRh3I-kl0F7bMez616tgM8Q/edit?usp=sharing I've also looked for prior work, and saw that Spark actually has something comparable: https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/sql/Row.html but when the options are finished it will be far more powerful as it is not limited on fields. _/ _/ Alex Van Boxel On Wed, Jan 29, 2020 at 4:55 AM Kenneth Knowles <[email protected]> wrote: > Using schema types for the metadata values is a nice touch. > > Are the options expected to be common across many fields? Perhaps the name > should be a URN to make it clear to be careful about collisions? (just a > synonym for "name" in practice, but with different connotation) > > I generally like this... but the examples (all but one) are weird things > that I don't really understand how they are done or who is responsible for > them. > > One way to go is this: if options are maybe not understood by all > consumers, then they can't really change behavior. Kind of like how URN and > payload on a composite transform can be ignored and just the expansion used. > > Kenn > > On Sun, Jan 26, 2020 at 8:27 AM Alex Van Boxel <[email protected]> wrote: > >> Hi everyone, >> >> I'm proud to announce my first real proposal. The proposal describes Beam >> Schema Options. This is an extension to the Schema API to add typed meta >> data to to Rows, Field and Logical Types: >> >> >> https://docs.google.com/document/d/1yCCRU5pViVQIO8-YAb66VRh3I-kl0F7bMez616tgM8Q/edit?usp=sharing >> >> To give you some context where this proposal comes from: We've been using >> dynamic meta driven pipelines for a while, but till now in an awkward and >> hacky way (see my talks at the previous Beam Summits). This proposal would >> bring a good way to work with metadata on the metadata :-). >> >> The proposal points to 2 pull requests with the implementation, one for >> the API another for translating proto options to beam options. >> >> Thanks to Brian Hulette and Reuven Lax for the initial feedback. All >> feedback is welcome. >> >> _/ >> _/ Alex Van Boxel >> >
