Re: Some extensions to the DoFn API

Thanks ! I will work on this one then ;) Regards JB On 04/06/2018 16:55, Reuven Lax wrote: > I'll file a JIRA to track the idea. > > On Mon, Jun 4, 2018 at 5:52 PM Jean-Baptiste Onofré > wrote: > > Exactly, that's why something like @xpath or @json-path could be

Re: Some extensions to the DoFn API

I'll file a JIRA to track the idea. On Mon, Jun 4, 2018 at 5:52 PM Jean-Baptiste Onofré wrote: > Exactly, that's why something like @xpath or @json-path could be > interesting. > > Regards > JB > > On 04/06/2018 16:48, Reuven Lax wrote: > > Interesting. And given that Beam Schemas are recursive

Re: Some extensions to the DoFn API

Exactly, that's why something like @xpath or @json-path could be interesting. Regards JB On 04/06/2018 16:48, Reuven Lax wrote: > Interesting. And given that Beam Schemas are recursive (a row can > contain nested rows), we might actually need something like xpath if we > want to make this fully

Re: Some extensions to the DoFn API

Interesting. And given that Beam Schemas are recursive (a row can contain nested rows), we might actually need something like xpath if we want to make this fully general. Reuven On Mon, Jun 4, 2018 at 5:45 PM Jean-Baptiste Onofré wrote: > Yup, it makes sense, it's what I had in mind. > > In

Re: Some extensions to the DoFn API

Yup, it makes sense, it's what I had in mind. In Apache Camel, in a Processor (similar to a DoFn), we can also pass directly languages to the arguments. We can imagine something like: @ProcessElement void process(@json-path("foo") String foo) @ProcessElement void process(@xpath("//foo") String

Re: Some extensions to the DoFn API

In the schema branch I have already added some annotations for Schema. However in the future I think we could go even further and allow users to pick individual fields out of the row schema. e.g. the user might have a Schema with 100 fields, but only want to process userId and geo location. I

Re: Some extensions to the DoFn API

Hi Reuven, That's a great improvement for user. I don't see an easy way to have annotation about side input/output. I think we can also plan some extension annotation about schema. Like @Element(schema = foo) in addition of the type. Thoughts ? Regards JB On 04/06/2018 16:06, Reuven Lax wrote: