Tushar, Can you create an alternate pull request with this change.
Thanks > On Apr 26, 2016, at 1:09 PM, Tushar Gosavi <[email protected]> wrote: > > Hi All, > > We could achieve this by adding a common interface between Operator and > Module without backward compatibility issue. I did a quick prototype with > this approach and was able to run > PiDemo application from Malhar built against previous version of Apex. With > this changes > module writer don't have to override lifecycle methods required by Operator. > > Let me know if this approach is workable. I will update the pull request > #313 with more polished > changes. > > Regards, > - Tushar. > > > On Wed, Apr 6, 2016 at 1:52 AM, Tushar Gosavi <[email protected]> > wrote: > >> Hi Thomas, >> >> I agree with you that module is a composite operator except its not >> present at the execution time. Actually I feel that there should be another >> Interface `Node` which specifies a node in the DAG, from which Operator and >> Module interfaces are extended. This way we may support transformation at >> logicalPlan level in future. Adding this will break the compatibility with >> current code. Hence my suggestion was to extend Module form Operator as it >> is considered as composite operator. >> >> Another motivation is avoid code duplication and work duplication to >> support modules in json format, in high level api or new future api's that >> will support adding module and operator to DAG. >> >> Currently we extract the port information using reflection, which can work >> regardless of interface hierarchy, but meta object associated with Port has >> reference to OperatorMeta. Changing it would result in many changes in >> LogicalPlan without any common interface. >> >> Thanks, >> -Tushar. >> >> >> On Tue, Mar 29, 2016 at 8:37 PM, Thomas Weise <[email protected]> >> wrote: >> >>> Tushar, >>> >>> I agree with the motivation of #2 but not with the specific changes you >>> suggest. "Module" is a composite operator from a users perspective, but >>> the >>> operator interface defines callbacks for the execution layer that are not >>> applicable to "Module". >>> >>> Also, ports are fields, they can be discovered regardless of interface >>> hierarchy. >>> >>> Thomas >>> >>> On Wed, Mar 23, 2016 at 3:55 AM, Tushar Gosavi <[email protected]> >>> wrote: >>> >>>> Hi All, >>>> >>>> I am planning provide support for adding modules into the json and >>> property >>>> file specification of DAG. We will go with the same syntax as discussed >>> in >>>> following mail thread. >>> https://mail-archives.apache.org/mod_mbox/incubator-apex-dev/201512.mbox/%3C565D0E2C.2000805%40datatorrent.com%3E >>>> >>>> >>>> There are multiple choices for the design and I want some suggestion >>> from >>>> the community aboutg how to go about adding this support. To support >>>> json/property format, Module should support following functionality >>>> - Property format uses setSource, addSink methods of StreamMeta, which >>> were >>>> not properly supported by Module. >>>> - Module need capability of extracting port object given names. Operator >>>> provides this functionality through Operators.describe. >>>> >>>> Approach 1) >>>> As module meta and operator meta shares common fields such as name, port >>>> information. We can separate this out in a common class NodeMeta and >>> derive >>>> OperatorMeta and ModuleMeta from it. This class will handle extracting >>> port >>>> information form the object. The changes involve are >>>> - Split OperatorMeta object >>>> - PortMeta objects will contain NodeMeta references than >>> OperatorMeta, >>>> in some places we will have to perform unchecked cast from NodeMeta to >>>> OperatorMeta. where code was expecting OperatorMeta from the >>>> PortMeta. >>>> - Support setSource and addSink methods. >>>> - Change Operators.describe to accept Module or Operator and return >>>> port mapping. >>>> - Need instanceof call at few places to check if object is Module or >>>> Operator, before calling specific API while working with property and >>> json >>>> file. >>>> - change signature of few method which accepts Operator to Object. >>> as >>>> Module and Operator do not share a common parent but requires >>>> common processing >>>> - Replace OperatorMeta with NodeMeta in most of the classes. >>>> >>>> >>>> Approach 2) >>>> Make Module extends Operator and make ModuleMeta extends OperatorMeta. >>> The >>>> changes involved will be >>>> - Change Module interface to extend Operator >>>> - add support for setSource and addSink for Modules. >>>> - addOperator will inspect type of object and call addModule if >>> operator >>>> is a module. >>>> >>>> Disadvantage >>>> - This will break compatibility but this should not be a problem as >>> Module >>>> is still an evolving API. >>>> - Operator lifecyle methods will not get executed for module and we can >>>> document this. >>>> >>>> Advantages >>>> - This will avoid code duplication at multiple places where there is >>>> similarity between Module and Operator, and there is lot of similarity >>>> while defining logical DAG. >>>> - This will also automatically add support for Module in api being >>>> developed for operator. For example high level api. >>>> - This will make module and operator interchangeable in application. >>>> >>>> I will prefer Approach 2. >>>> >>>> Regards, >>>> -Tushar. >> >>
