JIRA it is.. On Wed, Dec 7, 2016 at 10:09 AM, Sanjay Pujare <[email protected]> wrote:
> Should we continue the discussion in the JIRA? > > Making it generic the way you are suggesting – won’t it cause problems of > consistency etc e.g. different hooks do different things? Also are there a > use cases for such generic hooks to justify the effort involved? > > On 12/7/16, 7:29 AM, "Pramod Immaneni" <[email protected]> wrote: > > Tushar, > > Why specifically limit it to client side preparation of the DAG before > the > application is launched? Why not make it possible to have general hooks > that can apply even when the application is running in the different > distributed components of the application such as the containers and > stram. > The hooks could be registered to be asynchronous or synchronous. The > asynchronous ones could be handled via a bus, we already have a bus > called mbassador that we use today and it could potentially be used for > this. The entire functionality is akin to something like dtrace > without the > dynamic part. > > Thanks > > On Fri, Nov 25, 2016 at 3:24 AM, Tushar Gosavi <[email protected] > > > wrote: > > > Opened a Jira https://issues.apache.org/jira/browse/APEXCORE-577 > for this. > > > > - Tushar. > > > > > > On Mon, Nov 21, 2016 at 9:59 PM, Amol Kekre <[email protected]> > wrote: > > > Ananth, > > > The current API allows changing properties of a running app. The > new > > > proposed API is not needed to do so. > > > > > > Thks > > > Amol > > > > > > > > > On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi < > [email protected]> > > > wrote: > > > > > >> Hi Ananth, > > >> > > >> We can not change runtime properties through this API. The current > > >> flow of apex application execution is > > >> > > >> 1) StramClient prepares application, inject properties from > > >> properties.xml / user provided xml files and validates dag > > >> 2) StramClient copies required jars and serialized plan to HDFS > and > > >> launch master container. > > >> 3) Application master reads serialised plan from HDFS and starts > > >> deploying StramClient as per deployment plan. > > >> > > >> The visitor will examine the DAG in stage 1, hence visitor can > only > > >> change the initial state. The execution of the application is not > > >> affected. > > >> > > >> - Tushar. > > >> > > >> > > >> On Sat, Nov 19, 2016 at 3:40 AM, ananth <[email protected]> > wrote: > > >> > How does this work for the stateful operators ? Can we use this > to > > >> override > > >> > properties that are deserialized ? > > >> > > > >> > Regards, > > >> > > > >> > Ananth > > >> > > > >> > > > >> > > > >> > On 18/11/16 05:53, Tushar Gosavi wrote: > > >> >> > > >> >> The code will execute before application master is launched, > it is > > >> >> just one time activity during application startup. Few use > cases I > > >> >> could think are > > >> >> > > >> >> - Operator validation/configuration validator > > >> >> jdbc operator could check if database is accessible with > given > > >> >> credentials. > > >> >> file output operator could if directory exists and > filesystem is > > >> >> writable. > > >> >> > > >> >> - Injection of properties in operators from external sources. > > >> >> > > >> >> - If two operator wants to exchange some information based on > > >> >> configuration, they could do it through visitor. for example > > >> >> TUPLE_SCHEMA can be set on downstream operator port based on > > operators > > >> >> input TUPLE_SCHEMA and its configuration (for example > projection > > >> >> operator which drops few columns, could create a new class > with fewer > > >> >> fields and set it as tuple schema on downstream operator port). > > >> >> > > >> >> - For pojo enabled operator (port where TUPLE_SCHEMA is > defined), a > > >> >> efficient stream codec could be written using asm library for > > >> >> serialisation and use that as stream codec instead of default > one. > > >> >> > > >> >> -Tushar. > > >> >> > > >> >> > > >> >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare < > > [email protected] > > >> > > > >> >> wrote: > > >> >>> > > >> >>> There is a risk if the user written code blocks the thread or > > crashes > > >> the > > >> >>> process. What are the real life examples of this use case? > > >> >>> > > >> >>> > > >> >>> On 11/17/16, 9:21 AM, "amol kekre" <[email protected]> > wrote: > > >> >>> > > >> >>> +1. Opening up the API for users to put in their own > code is > > good. > > >> >>> In > > >> >>> general we should enable users to register their code in > a lot > > of > > >> >>> scenerios. > > >> >>> > > >> >>> Thks > > >> >>> Amol > > >> >>> > > >> >>> On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi > > >> >>> <[email protected]> > > >> >>> wrote: > > >> >>> > > >> >>> > Yes, It could happen after current DAG validation and > before > > the > > >> >>> > application master is launched. > > >> >>> > > > >> >>> > - Tushar. > > >> >>> > > > >> >>> > > > >> >>> > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath > > >> >>> <[email protected]> > > >> >>> > wrote: > > >> >>> > > When would the visits happen ? Just before normal > > validation ? > > >> >>> > > > > >> >>> > > Ram > > >> >>> > > > > >> >>> > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi > > >> >>> <[email protected]> > > >> >>> > wrote: > > >> >>> > > > > >> >>> > >> Hi All, > > >> >>> > >> > > >> >>> > >> How about adding visitor like API for DAG in Apex, > and an > > api > > >> >>> to > > >> >>> > >> register visitor for the DAG. > > >> >>> > >> Possible use cases are > > >> >>> > >> - Validator visitor which could validate the dag > > >> >>> > >> - Visitor to inject properties/attribute in the > > >> >>> operator/streams from > > >> >>> > >> some external sources. > > >> >>> > >> - Platform does not support validation of > individual > > >> >>> operators. > > >> >>> > >> developer could write a validator visitor which > would call > > >> >>> validate > > >> >>> > >> function of operator if it implements Validator > interface. > > >> >>> > >> - generate output schema based on operator config > and > > input > > >> >>> schema, > > >> >>> > >> and set the schema on output stream. > > >> >>> > >> > > >> >>> > >> Sample API : > > >> >>> > >> > > >> >>> > >> dag.registerVisitor(DAGVisitor visitor); > > >> >>> > >> > > >> >>> > >> Call order of visitorFunctions. > > >> >>> > >> - preVisitDAG(Attributes) // dag attributes > > >> >>> > >> for all operators > > >> >>> > >> - visitOperator(OperatorMeta meta) // access to > > operator, > > >> >>> name, > > >> >>> > >> attributes, properties > > >> >>> > >> ports > > >> >>> > >> - visitStream(StreamMeta meta) // access to > > >> >>> > >> stream/name/attributes/properties/ports > > >> >>> > >> - postVisitDAG() > > >> >>> > >> > > >> >>> > >> Regards, > > >> >>> > >> -Tushar. > > >> >>> > >> > > >> >>> > > > >> >>> > > >> >>> > > >> >>> > > >> > > > >> > > > > > >
