Re: Schema Discovery Support in Apex Applications

Will do that. On Fri, Feb 3, 2017 at 3:15 PM, Vlad Rozov wrote: > IMO, it will be good to summarize schema use case and proposed approach to > support it on the control tuple e-mail thread. Not everyone interested in > the custom control tuple may be following schema

Re: Schema Discovery Support in Apex Applications

IMO, it will be good to summarize schema use case and proposed approach to support it on the control tuple e-mail thread. Not everyone interested in the custom control tuple may be following schema support thread. Thank you, Vlad On 2/3/17 08:47, Pramod Immaneni wrote: On Fri, Feb 3, 2017

Re: Schema Discovery Support in Apex Applications

On Fri, Feb 3, 2017 at 7:59 AM, Thomas Weise wrote: > Agreed. As noted the main concern was the ability to support idempotency. > It isn't really "re-ordering" because when you have multiple input ports, > there isn't any ordering guarantee within a streaming window. > The

Re: Schema Discovery Support in Apex Applications

Agreed. As noted the main concern was the ability to support idempotency. It isn't really "re-ordering" because when you have multiple input ports, there isn't any ordering guarantee within a streaming window. The end window boundary is good when the control tuple needs to be processed after all

Re: Schema Discovery Support in Apex Applications

I second the proposal to revisit custom control tuple delivery and re-ordering. Schema support brings a use case that was missing when we discussed custom control tuples. Thank you, Vlad On 2/1/17 21:56, Pramod Immaneni wrote: This can be done neatly and possibly completely outside the

Re: Schema Discovery Support in Apex Applications

​I think the only concern for deciding the end-of-window-processing behavior for control tuples was that developers may, inadvertently mess up the idempotency of the operator. Even though we are trying to enforce this, it seems that even the emit of a control tuple (from the source) may not happen

Re: Schema Discovery Support in Apex Applications

This can be done neatly and possibly completely outside the engine if we are able to deliver schema information via the control tuple mechanism. Current control tuple proposal reorders the control tuple to be delivered at the end of the window to the operator. This would not be feasible for

Re: Schema Discovery Support in Apex Applications

Consumer of output port operator schema is going next downstream operator. On Tue, Jan 31, 2017 at 4:01 AM, Sergey Golovko wrote: > Sorry, I’m a new person in the APEX team. And I don't understand clearly > who are consumers of the output port operator schema(s). > > 1.

Re: Schema Discovery Support in Apex Applications

Sorry, I’m a new person in the APEX team. And I don't understand clearly who are consumers of the output port operator schema(s). 1. If the consumers are non-run-time callers like the application manager or UI designer, maybe it makes sense to use Java static method(s) to retrieve the output

Re: Schema Discovery Support in Apex Applications

Thank you all for the feedback. I've created a Jira for this: APEXCORE-623 and I'll attach the same document and link to this mailchain there. As a first part of this Jira, there are 2 steps I would like to propose: 1. Add following interface at com.datatorrent.common.util.SchemaAware.

Re: Schema Discovery Support in Apex Applications

+1 to have this feature. -Priyanka On Tue, Jan 17, 2017 at 9:18 PM, Pramod Immaneni wrote: > +1 > > On Mon, Jan 16, 2017 at 1:23 AM, Chinmay Kolhatkar > wrote: > > > Hi All, > > > > Currently a DAG that is generated by user, if contains any POJOfied

Re: Schema Discovery Support in Apex Applications

+1 On Mon, Jan 16, 2017 at 1:23 AM, Chinmay Kolhatkar wrote: > Hi All, > > Currently a DAG that is generated by user, if contains any POJOfied > operators, TUPLE_CLASS attribute needs to be set on each and every port > which receives or sends a POJO. > > For e.g., if a DAG

Re: Schema Discovery Support in Apex Applications

+1 Regards, Mohit On Tue, Jan 17, 2017 at 12:11 PM, Bhupesh Chawda wrote: > +1 for the feature. > > ~ Bhupesh > > On Mon, Jan 16, 2017 at 5:09 PM, Chinmay Kolhatkar > wrote: > > > Those are not really anonymous POJOs... The definition of POJO will

Re: Schema Discovery Support in Apex Applications

+1 for the feature. ~ Bhupesh On Mon, Jan 16, 2017 at 5:09 PM, Chinmay Kolhatkar wrote: > Those are not really anonymous POJOs... The definition of POJO will be > known to user as based on that only upstream operator will convey the tuple > type the operator will be

Re: Schema Discovery Support in Apex Applications

Those are not really anonymous POJOs... The definition of POJO will be known to user as based on that only upstream operator will convey the tuple type the operator will be emitting. Using that information user can configure the operators. Those properties will be a bit different though. On Mon,

Re: Schema Discovery Support in Apex Applications

+1 for the idea. I just had one question. As I understand, there will be some form of Anonymous POJO used as objects to pass information from one operator to another. Can you share how the user/operator developer would access the tuple object in case he wishes to do something with it? Ajay On

Schema Discovery Support in Apex Applications

Hi All, Currently a DAG that is generated by user, if contains any POJOfied operators, TUPLE_CLASS attribute needs to be set on each and every port which receives or sends a POJO. For e.g., if a DAG is like File -> Parser -> Transform -> Dedup -> Formatter -> Kafka, then TUPLE_CLASS attribute