Ananth, The current API allows changing properties of a running app. The new proposed API is not needed to do so.
Thks Amol On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi <[email protected]> wrote: > Hi Ananth, > > We can not change runtime properties through this API. The current > flow of apex application execution is > > 1) StramClient prepares application, inject properties from > properties.xml / user provided xml files and validates dag > 2) StramClient copies required jars and serialized plan to HDFS and > launch master container. > 3) Application master reads serialised plan from HDFS and starts > deploying StramClient as per deployment plan. > > The visitor will examine the DAG in stage 1, hence visitor can only > change the initial state. The execution of the application is not > affected. > > - Tushar. > > > On Sat, Nov 19, 2016 at 3:40 AM, ananth <[email protected]> wrote: > > How does this work for the stateful operators ? Can we use this to > override > > properties that are deserialized ? > > > > Regards, > > > > Ananth > > > > > > > > On 18/11/16 05:53, Tushar Gosavi wrote: > >> > >> The code will execute before application master is launched, it is > >> just one time activity during application startup. Few use cases I > >> could think are > >> > >> - Operator validation/configuration validator > >> jdbc operator could check if database is accessible with given > >> credentials. > >> file output operator could if directory exists and filesystem is > >> writable. > >> > >> - Injection of properties in operators from external sources. > >> > >> - If two operator wants to exchange some information based on > >> configuration, they could do it through visitor. for example > >> TUPLE_SCHEMA can be set on downstream operator port based on operators > >> input TUPLE_SCHEMA and its configuration (for example projection > >> operator which drops few columns, could create a new class with fewer > >> fields and set it as tuple schema on downstream operator port). > >> > >> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a > >> efficient stream codec could be written using asm library for > >> serialisation and use that as stream codec instead of default one. > >> > >> -Tushar. > >> > >> > >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <[email protected] > > > >> wrote: > >>> > >>> There is a risk if the user written code blocks the thread or crashes > the > >>> process. What are the real life examples of this use case? > >>> > >>> > >>> On 11/17/16, 9:21 AM, "amol kekre" <[email protected]> wrote: > >>> > >>> +1. Opening up the API for users to put in their own code is good. > >>> In > >>> general we should enable users to register their code in a lot of > >>> scenerios. > >>> > >>> Thks > >>> Amol > >>> > >>> On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi > >>> <[email protected]> > >>> wrote: > >>> > >>> > Yes, It could happen after current DAG validation and before the > >>> > application master is launched. > >>> > > >>> > - Tushar. > >>> > > >>> > > >>> > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath > >>> <[email protected]> > >>> > wrote: > >>> > > When would the visits happen ? Just before normal validation ? > >>> > > > >>> > > Ram > >>> > > > >>> > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi > >>> <[email protected]> > >>> > wrote: > >>> > > > >>> > >> Hi All, > >>> > >> > >>> > >> How about adding visitor like API for DAG in Apex, and an api > >>> to > >>> > >> register visitor for the DAG. > >>> > >> Possible use cases are > >>> > >> - Validator visitor which could validate the dag > >>> > >> - Visitor to inject properties/attribute in the > >>> operator/streams from > >>> > >> some external sources. > >>> > >> - Platform does not support validation of individual > >>> operators. > >>> > >> developer could write a validator visitor which would call > >>> validate > >>> > >> function of operator if it implements Validator interface. > >>> > >> - generate output schema based on operator config and input > >>> schema, > >>> > >> and set the schema on output stream. > >>> > >> > >>> > >> Sample API : > >>> > >> > >>> > >> dag.registerVisitor(DAGVisitor visitor); > >>> > >> > >>> > >> Call order of visitorFunctions. > >>> > >> - preVisitDAG(Attributes) // dag attributes > >>> > >> for all operators > >>> > >> - visitOperator(OperatorMeta meta) // access to operator, > >>> name, > >>> > >> attributes, properties > >>> > >> ports > >>> > >> - visitStream(StreamMeta meta) // access to > >>> > >> stream/name/attributes/properties/ports > >>> > >> - postVisitDAG() > >>> > >> > >>> > >> Regards, > >>> > >> -Tushar. > >>> > >> > >>> > > >>> > >>> > >>> > > >
