+1. Opening up the API for users to put in their own code is good. In general we should enable users to register their code in a lot of scenerios.
Thks Amol On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi <[email protected]> wrote: > Yes, It could happen after current DAG validation and before the > application master is launched. > > - Tushar. > > > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath <[email protected]> > wrote: > > When would the visits happen ? Just before normal validation ? > > > > Ram > > > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi <[email protected]> > wrote: > > > >> Hi All, > >> > >> How about adding visitor like API for DAG in Apex, and an api to > >> register visitor for the DAG. > >> Possible use cases are > >> - Validator visitor which could validate the dag > >> - Visitor to inject properties/attribute in the operator/streams from > >> some external sources. > >> - Platform does not support validation of individual operators. > >> developer could write a validator visitor which would call validate > >> function of operator if it implements Validator interface. > >> - generate output schema based on operator config and input schema, > >> and set the schema on output stream. > >> > >> Sample API : > >> > >> dag.registerVisitor(DAGVisitor visitor); > >> > >> Call order of visitorFunctions. > >> - preVisitDAG(Attributes) // dag attributes > >> for all operators > >> - visitOperator(OperatorMeta meta) // access to operator, name, > >> attributes, properties > >> ports > >> - visitStream(StreamMeta meta) // access to > >> stream/name/attributes/properties/ports > >> - postVisitDAG() > >> > >> Regards, > >> -Tushar. > >> >
