Of course! I think some BeamSQL folks should be involved as well, as this directly affects SQL work. Anton especially has expressed interest in Row and schemas.
Reuven On Mon, Mar 5, 2018 at 4:30 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Cool, > > can I work with you on this (sharing a branch for instance) ? > > Thanks ! > Regards > JB > > On 03/05/2018 01:01 PM, Reuven Lax wrote: > > Yes, I do have a PoC in progress. The Beam Row class was being > refactored, so I > > paused to wait for that to finish. > > > > > > On Sun, Mar 4, 2018 at 8:24 PM Jean-Baptiste Onofré <j...@nanthrax.net > > <mailto:j...@nanthrax.net>> wrote: > > > > Hi Reuven, > > > > I revive this discussion as I think it would be a great addition. > > > > We had discussion on the fly, but I think now, as base for > discussion, it would > > be great to have a feature branch where we can start some > sketch/impl and > > discuss. > > > > @Reuven, did you start a PoC with what you proposed: > > - SchemaCoder > > - SchemaRegistry > > - @FieldAccess on DoFn > > - Select.fields PTransform > > ? > > > > If not, I'm volunteer to start the branch and start to sketch. > > > > Thoughts ? > > > > Regards > > JB > > > > On 02/04/2018 08:23 PM, Reuven Lax wrote: > > > Cool, let's chat about this on slack for a bit (which I realized > I've been > > > signed out of for some time). > > > > > > Reuven > > > > > > On Sun, Feb 4, 2018 at 9:21 AM, Jean-Baptiste Onofré < > j...@nanthrax.net > > <mailto:j...@nanthrax.net> > > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote: > > > > > > Sorry guys, I was off today. Happy to be part of the party too > ;) > > > > > > Regards > > > JB > > > > > > On 02/04/2018 06:19 PM, Reuven Lax wrote: > > > > Romain, since you're interested maybe the two of us should > put > > together a > > > > proposal for how to set this things (hints, schema) on > PCollections? > > I don't > > > > think it'll be hard - the previous list thread on hints > already > > agreed on a > > > > general approach, and we would just need to flesh it out. > > > > > > > > BTW in the past when I looked, Json schemas seemed to have > some odd > > limitations > > > > inherited from Javascript (e.g. no distinction between > integer and > > > > floating-point types). Is that still true? > > > > > > > > Reuven > > > > > > > > On Sun, Feb 4, 2018 at 9:12 AM, Romain Manni-Bucau > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>> > wrote: > > > > > > > > > > > > > > > > 2018-02-04 17:53 GMT+01:00 Reuven Lax <re...@google.com > > <mailto:re...@google.com> <mailto:re...@google.com <mailto: > re...@google.com>> > > > > <mailto:re...@google.com <mailto:re...@google.com> > > <mailto:re...@google.com <mailto:re...@google.com>>>>: > > > > > > > > > > > > > > > > On Sun, Feb 4, 2018 at 8:42 AM, Romain Manni-Bucau > > > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>> > wrote: > > > > > > > > > > > > 2018-02-04 17:37 GMT+01:00 Reuven Lax < > re...@google.com > > <mailto:re...@google.com> <mailto:re...@google.com <mailto: > re...@google.com>> > > > > <mailto:re...@google.com <mailto: > re...@google.com> > > <mailto:re...@google.com <mailto:re...@google.com>>>>: > > > > > > > > I'm not sure where proto comes from here. > Proto is > > one example > > > > of a type that has a schema, but only one > example. > > > > > > > > 1. In the initial prototype I want to avoid > > modifying the > > > > PCollection API. So I think it's best to > create a > > special > > > > SchemaCoder, and pass the schema into this > coder. > > Later we > > > might > > > > targeted APIs for this instead of going > through a coder. > > > > 1.a I don't see what hints have to do with > this? > > > > > > > > > > > > Hints are a way to replace the new API and unify > the way > > to pass > > > > metadata in beam instead of adding a new custom > way each > > time. > > > > > > > > > > > > I don't think schema is a hint. But I hear what your > saying > > - hint > > > is a > > > > type of PCollection metadata as is schema, and we > should have a > > > unified > > > > API for setting such metadata. > > > > > > > > > > > > :), Ismael pointed me out earlier this week that "hint" > had an > > old meaning > > > > in beam. My usage is purely the one done in most EE spec > (your > > > "metadata" in > > > > previous answer). But guess we are aligned on the > meaning now, > > just wanted > > > > to be sure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. BeamSQL already has a generic record type > which fits > > > this use > > > > case very well (though we might modify it). > However as > > > mentioned > > > > in the doc, the user is never forced to use > this generic > > > record > > > > type. > > > > > > > > > > > > Well yes and not. A type already exists but 1. > it is > > very strictly > > > > limited (flat/columns only which is very few of > what big > > data SQL > > > > can do) and 2. it must be aligned on the > converge of > > generic data > > > > the schema will bring (really read "aligned" as > "dropped > > in favor > > > > of" - deprecated being a smooth way to do it). > > > > > > > > > > > > As I said the existing class needs to be modified > and extended, > > > and not > > > > just for this schema us was. It was meant to > represent > > Calcite SQL > > > rows, > > > > but doesn't quite even do that yet (Calcite supports > nested > > rows). > > > > However I think it's the right basis to start from. > > > > > > > > > > > > Agree on the state. Current impl issues I hit > (additionally to > > the nested > > > > support which would require by itself a kind of visitor > > solution) are the > > > > fact to own the schema in the record and handle field by > field the > > > > serialization instead of as a whole which is how it > would be handled > > > with a > > > > schema IMHO. > > > > > > > > Concretely what I don't want is to do a PoC which works > - they > > all work > > > > right? and integrate to beam without thinking to a global > > solution for > > > this > > > > generic record issue and its schema standardization. > This is where > > > Json(-P) > > > > has a lot of value IMHO but requires a bit more love > than just > > adding > > > schema > > > > in the model. > > > > > > > > > > > > > > > > > > > > > > > > So long story short the main work of this schema > track > > is not only > > > > on using schema in runners and other ways but > also > > starting to > > > make > > > > beam consistent with itself which is probably > the most > > important > > > > outcome since it is the user facing side of this > work. > > > > > > > > > > > > > > > > On Sun, Feb 4, 2018 at 12:22 AM, Romain > Manni-Bucau > > > > <rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com>> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>> > wrote: > > > > > > > > @Reuven: is the proto only about passing > schema > > or also the > > > > generic type? > > > > > > > > There are 2.5 topics to solve this issue: > > > > > > > > 1. How to pass schema > > > > 1.a. hints? > > > > 2. What is the generic record type > associated to > > a schema > > > > and how to express a schema relatively > to it > > > > > > > > I would be happy to help on 1.a and 2 > somehow if > > you need. > > > > > > > > Le 4 févr. 2018 03:30, "Reuven Lax" > > <re...@google.com <mailto:re...@google.com> <mailto:re...@google.com > > <mailto:re...@google.com>> > > > > <mailto:re...@google.com > > <mailto:re...@google.com> <mailto:re...@google.com > > <mailto:re...@google.com>>>> a > > > écrit : > > > > > > > > One more thing. If anyone here has > > experience with > > > > various OSS metadata stores (e.g. > Kafka > > Schema Registry > > > > is one example), would you like to > > collaborate on > > > > implementation? I want to make sure > that > > source schemas > > > > can be stored in a variety of OSS > metadata > > stores, and > > > > be easily pulled into a Beam > pipeline. > > > > > > > > Reuven > > > > > > > > On Sat, Feb 3, 2018 at 6:28 PM, > Reuven Lax > > > > <re...@google.com <mailto: > re...@google.com> > > <mailto:re...@google.com <mailto:re...@google.com>> <mailto: > re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>>>> wrote: > > > > > > > > Hi all, > > > > > > > > If there are no concerns, I > would like > > to start > > > > working on a prototype. It's > just a > > prototype, so I > > > > don't think it will have the > final API > > (e.g. for the > > > > prototype I'm going to avoid > change the > > API of > > > > PCollection, and use a "special" > Coder > > instead). > > > > Also even once we go beyond > prototype, > > it will be > > > > @Experimental for some time, so > the API > > will not be > > > > fixed in stone. > > > > > > > > Any more comments on this > approach > > before we start > > > > implementing a prototype? > > > > > > > > Reuven > > > > > > > > On Wed, Jan 31, 2018 at 1:12 PM, > Romain > > Manni-Bucau > > > > <rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com>> > > > > <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> <mailto:rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com>>>> wrote: > > > > > > > > If you need help on the json > part > > I'm happy to > > > > help. To give a few hints on > what is > > very > > > > doable: we can add an avro > module to > > johnzon > > > > (asf json{p,b} impl) to back > jsonp > > by avro > > > > (guess it will be one of the > first > > to be asked) > > > > for instance. > > > > > > > > > > > > Romain Manni-Bucau > > > > @rmannibucau > > > <https://twitter.com/rmannibucau < > https://twitter.com/rmannibucau>> | > > > > Blog < > https://rmannibucau.metawerx.net/ > > > <https://rmannibucau.metawerx.net/>> | Old > > > > Blog < > http://rmannibucau.wordpress.com > > > <http://rmannibucau.wordpress.com>> | Github > > > > < > https://github.com/rmannibucau > > > <https://github.com/rmannibucau>> | LinkedIn > > > > < > https://www.linkedin.com/in/rmannibucau > > > <https://www.linkedin.com/in/rmannibucau>> > > > > > > > > 2018-01-31 22:06 GMT+01:00 > Reuven Lax > > > > <re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>> > > <mailto:re...@google.com <mailto:re...@google.com> <mailto: > re...@google.com > > <mailto:re...@google.com>>>>: > > > > > > > > Agree. The initial > > implementation will be a > > > > prototype. > > > > > > > > On Wed, Jan 31, 2018 at > 12:21 PM, > > > > Jean-Baptiste Onofré > > <j...@nanthrax.net <mailto:j...@nanthrax.net> <mailto:j...@nanthrax.net > > <mailto:j...@nanthrax.net>> > > > > <mailto:j...@nanthrax.net > > <mailto:j...@nanthrax.net> > > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>>> wrote: > > > > > > > > Hi Reuven, > > > > > > > > Agree to be able to > describe the > > > schema > > > > with different > format. The > > good point > > > > about json schemas > is that > > they are > > > > described by a spec. > My point is > > > also to > > > > avoid the reinvent > the > > wheel. Just an > > > > abstract to be able > to use > > Avro, Json, > > > > Calcite, custom > schema > > descriptors > > > would > > > > be great. > > > > > > > > Using coder to > describe a schema > > > sounds > > > > like a smart move to > implement > > > quickly. > > > > However, it has to > be clear > > in term of > > > > documentation to > avoid "side > > > effect". I > > > > still think > > PCollection.setSchema() is > > > > better: it should be > > metadata (or hint > > > > ;))) on the > PCollection. > > > > > > > > Regards > > > > JB > > > > > > > > On 31/01/2018 20:16, > Reuven > > Lax wrote: > > > > > > > > As to the > question of > > how a schema > > > > should be > specified, I > > want to > > > > support several > common > > schema > > > > formats. So if a > user > > has a Json > > > > schema, or an > Avro > > schema, or a > > > > Calcite schema, > etc. there > > > should be > > > > adapters that > allow > > setting a > > > schema > > > > from any of > them. I > > don't think we > > > > should prefer > one over > > the other. > > > > While Romain is > right > > that many > > > > people know > Json, I > > think far > > > fewer > > > > people know Json > schemas. > > > > > > > > Agree, schemas > should not be > > > > enforced (for > one thing, > > that > > > > wouldn't be > backwards > > > compatible!). > > > > I think for the > initial > > > prototype I > > > > will probably > use a special > > > coder to > > > > represent the > schema (with > > > setSchema > > > > an option on the > coder), > > largely > > > > because it > doesn't require > > > modifying > > > > PCollection. > However I think > > > longer > > > > term a schema > should be an > > > optional > > > > piece of > metadata on the > > > PCollection > > > > object. Similar > to the > > previous > > > > discussion about > > "hints," I think > > > > this can be set > on the > > producing > > > > PTransform, and > a SetSchema > > > > PTransform will > allow > > attaching a > > > > schema to any > > PCollection (i.e. > > > > > > pc.apply(SetSchema.of(schema))). > > > > This part isn't > designed > > yet, > > > but I > > > > think schema > should be > > similar to > > > > hints, it's just > another > > piece of > > > > metadata on the > PCollection > > > (though > > > > something > interpreted by the > > > model, > > > > where hints are > > interpreted by the > > > > runner) > > > > > > > > Reuven > > > > > > > > On Tue, Jan 30, > 2018 at > > 1:37 AM, > > > > Jean-Baptiste > Onofré > > > > <j...@nanthrax.net > > <mailto:j...@nanthrax.net> > > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>> > > > > <mailto: > j...@nanthrax.net > > <mailto:j...@nanthrax.net> > > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> > > > > <mailto: > j...@nanthrax.net > > <mailto:j...@nanthrax.net> > > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>> > > > > <mailto: > j...@nanthrax.net > > <mailto:j...@nanthrax.net> > > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>>>> wrote: > > > > > > > > Hi, > > > > > > > > I think we > should > > avoid to mix > > > > two things in > the discussion > > > (and so > > > > the > document): > > > > > > > > 1. The > element of the > > > collection > > > > and the schema > itself > > are two > > > > different > things. > > > > By essence, > Beam > > should not > > > > enforce any > schema. > > That's why > > > I think > > > > it's a good > > > > idea to set > the schema > > > > optionally on the > > PCollection > > > > > > (pcollection.setSchema()). > > > > > > > > 2. From > point 1 > > comes two > > > > questions: how > do we > > represent a > > > > schema ? > > > > How can we > > > > leverage the > schema to > > > simplify > > > > the > serialization of the > > > element in the > > > > PCollection > and > > query ? These > > > > two questions > are not > > directly > > > related. > > > > > > > > 2.1 How do > we > > represent > > > the schema > > > > Json Schema > is a very > > > > interesting > idea. It > > could be an > > > > abstract and > > > > other > > > > providers, > like > > Avro, can be > > > > bind on it. It's > part of > > the json > > > > processing > spec > > > > (javax). > > > > > > > > 2.2. How > do we > > leverage the > > > > schema for query > and > > serialization > > > > Also in the > spec, > > json pointer > > > > is interesting > for the > > querying. > > > > Regarding the > > > > > serialization, > > jackson or > > > other > > > > data binder can > be used. > > > > > > > > It's still > rough > > ideas in my > > > > mind, but I like > > Romain's idea > > > about > > > > json-p usage. > > > > > > > > Once 2.3.0 > release > > is out, I > > > > will start to > update the > > > document with > > > > those ideas, > > > > and PoC. > > > > > > > > Thanks ! > > > > Regards > > > > JB > > > > > > > > On > 01/30/2018 08:42 > > AM, Romain > > > > Manni-Bucau > wrote: > > > > > > > > > > > > > > > Le 30 > janv. 2018 > > 01:09, > > > > "Reuven Lax" > > <re...@google.com <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>> > > > > <mailto: > re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>>> > > > > <mailto: > re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>> > > > > <mailto: > re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>>>> > > > > > > > > <mailto:re...@google.com <mailto:re...@google.com> > > <mailto:re...@google.com <mailto:re...@google.com>> > > > > <mailto: > re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>>> > > > > <mailto: > re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>> > > > > <mailto: > re...@google.com > > <mailto:re...@google.com> > > > <mailto:re...@google.com <mailto:re...@google.com>>>>>> a > écrit : > > > > > > > > > > > > > > > > > > > > On > Mon, Jan > > 29, 2018 at > > > > 12:17 PM, Romain > Manni-Bucau > > > > < > rmannibu...@gmail.com > > <mailto:rmannibu...@gmail.com> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>> > > > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com > >>>> > > > > > > > > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>> > > > > > > > > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> > > > > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> > > > <mailto:rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>>>>>> > wrote: > > > > > > > > > > Hi > > > > > > > > > > I > have some > > > questions > > > > on this: how > hierarchic > > schemas > > > > would work? > Seems > > > > > > it is not > > really > > > > supported by the > > ecosystem (out of > > > > custom > stuff) :(. > > > > > > How would it > > > > integrate > smoothly with > > other > > > > generic record > > > > types - N > bridges? > > > > > > > > > > > > > > > Do > you mean > > nested > > > > schemas? What do > you > > mean here? > > > > > > > > > > > > > > > Yes, > sorry - > > wrote the mail > > > > too late ;). Was > hierarchic > > > data and > > > > nested > schemas. > > > > > > > > > > > > > > > > > Concretely I wonder > > > > if using json > API couldnt be > > > > beneficial: > json-p is a > > > > > > nice generic > > > > abstraction with > a built in > > > querying > > > > mecanism > (jsonpointer) > > > > > > but no actual > > > > serialization > (even if > > json and > > > > binary json > > > > are very > > > > > > natural). > > The big > > > > advantage is to > have a > > well known > > > > ecosystem - > who > > > > > > doesnt > > know json > > > > today? - that > beam can reuse > > > for free: > > > > JsonObject > > > > > > (guess we > > dont want > > > > JsonValue > abstraction) > > for the > > > record > > > > type, > > > > > > > jsonschema standard > > > > for the schema, > jsonpointer > > > for the > > > > > > > > delection/projection > > > > etc... It doesnt > enforce the > > > actual > > > > serialization > > > > > > (json, > > smile, avro, > > > > ...) but provide > an > > expressive and > > > > alread known > API > > > > > > so i see > > it as > > > a big > > > > win-win for > users (no > > need to > > > learn > > > > a new API and > > > > > > use N bridges > > > in all > > > > ways) and beam > (impls > > are here and > > > > API design > > > > > > already > > thought). > > > > > > > > > > > > > > > I > assume > > you're talking > > > > about the API for > > setting schemas, > > > > not using > them. > > > > > Json > has many > > downsides > > > > and I'm not sure > it's > > true that > > > > everyone > knows it; > > > > > there > are also > > > competing > > > > schema APIs, > such as > > Avro etc.. > > > > However I > think we > > > > > > should give > > Json a fair > > > >
