Opened a Jira https://issues.apache.org/jira/browse/APEXCORE-577 for this.
- Tushar. On Mon, Nov 21, 2016 at 9:59 PM, Amol Kekre <[email protected]> wrote: > Ananth, > The current API allows changing properties of a running app. The new > proposed API is not needed to do so. > > Thks > Amol > > > On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi <[email protected]> > wrote: > >> Hi Ananth, >> >> We can not change runtime properties through this API. The current >> flow of apex application execution is >> >> 1) StramClient prepares application, inject properties from >> properties.xml / user provided xml files and validates dag >> 2) StramClient copies required jars and serialized plan to HDFS and >> launch master container. >> 3) Application master reads serialised plan from HDFS and starts >> deploying StramClient as per deployment plan. >> >> The visitor will examine the DAG in stage 1, hence visitor can only >> change the initial state. The execution of the application is not >> affected. >> >> - Tushar. >> >> >> On Sat, Nov 19, 2016 at 3:40 AM, ananth <[email protected]> wrote: >> > How does this work for the stateful operators ? Can we use this to >> override >> > properties that are deserialized ? >> > >> > Regards, >> > >> > Ananth >> > >> > >> > >> > On 18/11/16 05:53, Tushar Gosavi wrote: >> >> >> >> The code will execute before application master is launched, it is >> >> just one time activity during application startup. Few use cases I >> >> could think are >> >> >> >> - Operator validation/configuration validator >> >> jdbc operator could check if database is accessible with given >> >> credentials. >> >> file output operator could if directory exists and filesystem is >> >> writable. >> >> >> >> - Injection of properties in operators from external sources. >> >> >> >> - If two operator wants to exchange some information based on >> >> configuration, they could do it through visitor. for example >> >> TUPLE_SCHEMA can be set on downstream operator port based on operators >> >> input TUPLE_SCHEMA and its configuration (for example projection >> >> operator which drops few columns, could create a new class with fewer >> >> fields and set it as tuple schema on downstream operator port). >> >> >> >> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a >> >> efficient stream codec could be written using asm library for >> >> serialisation and use that as stream codec instead of default one. >> >> >> >> -Tushar. >> >> >> >> >> >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <[email protected] >> > >> >> wrote: >> >>> >> >>> There is a risk if the user written code blocks the thread or crashes >> the >> >>> process. What are the real life examples of this use case? >> >>> >> >>> >> >>> On 11/17/16, 9:21 AM, "amol kekre" <[email protected]> wrote: >> >>> >> >>> +1. Opening up the API for users to put in their own code is good. >> >>> In >> >>> general we should enable users to register their code in a lot of >> >>> scenerios. >> >>> >> >>> Thks >> >>> Amol >> >>> >> >>> On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi >> >>> <[email protected]> >> >>> wrote: >> >>> >> >>> > Yes, It could happen after current DAG validation and before the >> >>> > application master is launched. >> >>> > >> >>> > - Tushar. >> >>> > >> >>> > >> >>> > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath >> >>> <[email protected]> >> >>> > wrote: >> >>> > > When would the visits happen ? Just before normal validation ? >> >>> > > >> >>> > > Ram >> >>> > > >> >>> > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi >> >>> <[email protected]> >> >>> > wrote: >> >>> > > >> >>> > >> Hi All, >> >>> > >> >> >>> > >> How about adding visitor like API for DAG in Apex, and an api >> >>> to >> >>> > >> register visitor for the DAG. >> >>> > >> Possible use cases are >> >>> > >> - Validator visitor which could validate the dag >> >>> > >> - Visitor to inject properties/attribute in the >> >>> operator/streams from >> >>> > >> some external sources. >> >>> > >> - Platform does not support validation of individual >> >>> operators. >> >>> > >> developer could write a validator visitor which would call >> >>> validate >> >>> > >> function of operator if it implements Validator interface. >> >>> > >> - generate output schema based on operator config and input >> >>> schema, >> >>> > >> and set the schema on output stream. >> >>> > >> >> >>> > >> Sample API : >> >>> > >> >> >>> > >> dag.registerVisitor(DAGVisitor visitor); >> >>> > >> >> >>> > >> Call order of visitorFunctions. >> >>> > >> - preVisitDAG(Attributes) // dag attributes >> >>> > >> for all operators >> >>> > >> - visitOperator(OperatorMeta meta) // access to operator, >> >>> name, >> >>> > >> attributes, properties >> >>> > >> ports >> >>> > >> - visitStream(StreamMeta meta) // access to >> >>> > >> stream/name/attributes/properties/ports >> >>> > >> - postVisitDAG() >> >>> > >> >> >>> > >> Regards, >> >>> > >> -Tushar. >> >>> > >> >> >>> > >> >>> >> >>> >> >>> >> > >>
