Will do that.
On Fri, Feb 3, 2017 at 3:15 PM, Vlad Rozov wrote:
> IMO, it will be good to summarize schema use case and proposed approach to
> support it on the control tuple e-mail thread. Not everyone interested in
> the custom control tuple may be following schema support thread.
>
> Thank yo
IMO, it will be good to summarize schema use case and proposed approach
to support it on the control tuple e-mail thread. Not everyone
interested in the custom control tuple may be following schema support
thread.
Thank you,
Vlad
On 2/3/17 08:47, Pramod Immaneni wrote:
On Fri, Feb 3, 2017 a
On Fri, Feb 3, 2017 at 7:59 AM, Thomas Weise wrote:
> Agreed. As noted the main concern was the ability to support idempotency.
> It isn't really "re-ordering" because when you have multiple input ports,
> there isn't any ordering guarantee within a streaming window.
>
The reordering I was refer
Agreed. As noted the main concern was the ability to support idempotency.
It isn't really "re-ordering" because when you have multiple input ports,
there isn't any ordering guarantee within a streaming window.
The end window boundary is good when the control tuple needs to be
processed after all a
I second the proposal to revisit custom control tuple delivery and
re-ordering. Schema support brings a use case that was missing when we
discussed custom control tuples.
Thank you,
Vlad
On 2/1/17 21:56, Pramod Immaneni wrote:
This can be done neatly and possibly completely outside the engin
I think the only concern for deciding the end-of-window-processing
behavior for control tuples was that developers may, inadvertently mess up
the idempotency of the operator.
Even though we are trying to enforce this, it seems that even the emit of a
control tuple (from the source) may not happen
This can be done neatly and possibly completely outside the engine if we
are able to deliver schema information via the control tuple mechanism.
Current control tuple proposal reorders the control tuple to be delivered
at the end of the window to the operator. This would not be feasible for
schemas
I think dynamic schema would be good to consider (schema known and possibly
changing at runtime). Some applications cannot be written under the
assumption that the schema is known upfront.
Also, does this really need to leak into the engine? I think it would be
good to consider alternatives and tr
Consumer of output port operator schema is going next downstream operator.
On Tue, Jan 31, 2017 at 4:01 AM, Sergey Golovko
wrote:
> Sorry, I’m a new person in the APEX team. And I don't understand clearly
> who are consumers of the output port operator schema(s).
>
> 1. If the consumers are non
Sorry, Iâm a new person in the APEX team. And I don't understand clearly who
are consumers of the output port operator schema(s).
1. If the consumers are non-run-time callers like the application manager or UI
designer, maybe it makes sense to use Java static method(s) to retrieve the
output
Thank you all for the feedback.
I've created a Jira for this: APEXCORE-623 and I'll attach the same
document and link to this mailchain there.
As a first part of this Jira, there are 2 steps I would like to propose:
1. Add following interface at com.datatorrent.common.util.SchemaAware.
interface
+1 to have this feature.
-Priyanka
On Tue, Jan 17, 2017 at 9:18 PM, Pramod Immaneni
wrote:
> +1
>
> On Mon, Jan 16, 2017 at 1:23 AM, Chinmay Kolhatkar
> wrote:
>
> > Hi All,
> >
> > Currently a DAG that is generated by user, if contains any POJOfied
> > operators, TUPLE_CLASS attribute needs t
+1
On Mon, Jan 16, 2017 at 1:23 AM, Chinmay Kolhatkar
wrote:
> Hi All,
>
> Currently a DAG that is generated by user, if contains any POJOfied
> operators, TUPLE_CLASS attribute needs to be set on each and every port
> which receives or sends a POJO.
>
> For e.g., if a DAG is like File -> Parser
+1
Regards,
Mohit
On Tue, Jan 17, 2017 at 12:11 PM, Bhupesh Chawda
wrote:
> +1 for the feature.
>
> ~ Bhupesh
>
> On Mon, Jan 16, 2017 at 5:09 PM, Chinmay Kolhatkar
> wrote:
>
> > Those are not really anonymous POJOs... The definition of POJO will be
> > known to user as based on that only ups
+1 for the feature.
~ Bhupesh
On Mon, Jan 16, 2017 at 5:09 PM, Chinmay Kolhatkar
wrote:
> Those are not really anonymous POJOs... The definition of POJO will be
> known to user as based on that only upstream operator will convey the tuple
> type the operator will be emitting.
> Using that infor
Those are not really anonymous POJOs... The definition of POJO will be
known to user as based on that only upstream operator will convey the tuple
type the operator will be emitting.
Using that information user can configure the operators. Those properties
will be a bit different though.
On Mon, J
+1 for the idea.
I just had one question.
As I understand, there will be some form of Anonymous POJO used as objects
to pass information from one operator to another. Can you share how the
user/operator developer would access the tuple object in case he wishes to
do something with it?
Ajay
On
Hi All,
Currently a DAG that is generated by user, if contains any POJOfied
operators, TUPLE_CLASS attribute needs to be set on each and every port
which receives or sends a POJO.
For e.g., if a DAG is like File -> Parser -> Transform -> Dedup ->
Formatter -> Kafka, then TUPLE_CLASS attribute nee
18 matches
Mail list logo