The code will execute before application master is launched, it is just one time activity during application startup. Few use cases I could think are
- Operator validation/configuration validator jdbc operator could check if database is accessible with given credentials. file output operator could if directory exists and filesystem is writable. - Injection of properties in operators from external sources. - If two operator wants to exchange some information based on configuration, they could do it through visitor. for example TUPLE_SCHEMA can be set on downstream operator port based on operators input TUPLE_SCHEMA and its configuration (for example projection operator which drops few columns, could create a new class with fewer fields and set it as tuple schema on downstream operator port). - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a efficient stream codec could be written using asm library for serialisation and use that as stream codec instead of default one. -Tushar. On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <[email protected]> wrote: > There is a risk if the user written code blocks the thread or crashes the > process. What are the real life examples of this use case? > > > On 11/17/16, 9:21 AM, "amol kekre" <[email protected]> wrote: > > +1. Opening up the API for users to put in their own code is good. In > general we should enable users to register their code in a lot of > scenerios. > > Thks > Amol > > On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi <[email protected]> > wrote: > > > Yes, It could happen after current DAG validation and before the > > application master is launched. > > > > - Tushar. > > > > > > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath > <[email protected]> > > wrote: > > > When would the visits happen ? Just before normal validation ? > > > > > > Ram > > > > > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi <[email protected]> > > wrote: > > > > > >> Hi All, > > >> > > >> How about adding visitor like API for DAG in Apex, and an api to > > >> register visitor for the DAG. > > >> Possible use cases are > > >> - Validator visitor which could validate the dag > > >> - Visitor to inject properties/attribute in the operator/streams > from > > >> some external sources. > > >> - Platform does not support validation of individual operators. > > >> developer could write a validator visitor which would call validate > > >> function of operator if it implements Validator interface. > > >> - generate output schema based on operator config and input schema, > > >> and set the schema on output stream. > > >> > > >> Sample API : > > >> > > >> dag.registerVisitor(DAGVisitor visitor); > > >> > > >> Call order of visitorFunctions. > > >> - preVisitDAG(Attributes) // dag attributes > > >> for all operators > > >> - visitOperator(OperatorMeta meta) // access to operator, name, > > >> attributes, properties > > >> ports > > >> - visitStream(StreamMeta meta) // access to > > >> stream/name/attributes/properties/ports > > >> - postVisitDAG() > > >> > > >> Regards, > > >> -Tushar. > > >> > > > > >
