How does this work for the stateful operators ? Can we use this to
override properties that are deserialized ?
Regards,
Ananth
On 18/11/16 05:53, Tushar Gosavi wrote:
The code will execute before application master is launched, it is
just one time activity during application startup. Few use cases I
could think are
- Operator validation/configuration validator
jdbc operator could check if database is accessible with given credentials.
file output operator could if directory exists and filesystem is writable.
- Injection of properties in operators from external sources.
- If two operator wants to exchange some information based on
configuration, they could do it through visitor. for example
TUPLE_SCHEMA can be set on downstream operator port based on operators
input TUPLE_SCHEMA and its configuration (for example projection
operator which drops few columns, could create a new class with fewer
fields and set it as tuple schema on downstream operator port).
- For pojo enabled operator (port where TUPLE_SCHEMA is defined), a
efficient stream codec could be written using asm library for
serialisation and use that as stream codec instead of default one.
-Tushar.
On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <[email protected]> wrote:
There is a risk if the user written code blocks the thread or crashes the
process. What are the real life examples of this use case?
On 11/17/16, 9:21 AM, "amol kekre" <[email protected]> wrote:
+1. Opening up the API for users to put in their own code is good. In
general we should enable users to register their code in a lot of
scenerios.
Thks
Amol
On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi <[email protected]>
wrote:
> Yes, It could happen after current DAG validation and before the
> application master is launched.
>
> - Tushar.
>
>
> On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath <[email protected]>
> wrote:
> > When would the visits happen ? Just before normal validation ?
> >
> > Ram
> >
> > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi <[email protected]>
> wrote:
> >
> >> Hi All,
> >>
> >> How about adding visitor like API for DAG in Apex, and an api to
> >> register visitor for the DAG.
> >> Possible use cases are
> >> - Validator visitor which could validate the dag
> >> - Visitor to inject properties/attribute in the operator/streams from
> >> some external sources.
> >> - Platform does not support validation of individual operators.
> >> developer could write a validator visitor which would call validate
> >> function of operator if it implements Validator interface.
> >> - generate output schema based on operator config and input schema,
> >> and set the schema on output stream.
> >>
> >> Sample API :
> >>
> >> dag.registerVisitor(DAGVisitor visitor);
> >>
> >> Call order of visitorFunctions.
> >> - preVisitDAG(Attributes) // dag attributes
> >> for all operators
> >> - visitOperator(OperatorMeta meta) // access to operator, name,
> >> attributes, properties
> >> ports
> >> - visitStream(StreamMeta meta) // access to
> >> stream/name/attributes/properties/ports
> >> - postVisitDAG()
> >>
> >> Regards,
> >> -Tushar.
> >>
>
