It is relatively easy to describe the justification for this change without getting into the weeds and hairsplitting words.
A DAG is built not only to launch an application but also to let a user visualize and configure it. Currently "populateDAG" is the only method we require application writers to implement and they implement it with the goal of running the application. So it can use properties, configuration and code that is really only needed if you want to "run" the DAG. As mentioned above a perfectly valid use case is that a platform allows a user to construct a DAG, visualize it and then attach configuration values to various components in the DAG, save these values as some kind of a "configuration package" and then at a future date run the DAG with this setup. This is consistent with the view that construction of a pipeline and execution of the pipeline are 2 separate phases and should be delineated as such. If you understand and agree with the justification we can work on improving the original proposal. Sanjay On Thu, Dec 21, 2017 at 8:27 AM, Vlad Rozov <vro...@apache.org> wrote: > "Sometimes" is not a use case. Config is not a context. > > Without concrete use cases the proposed change is not well justified. > populateDAG() is supposed to populate DAG, not to record anything in an > external system. It was a design goal for plugins. > > Thank you, > > Vlad > > > On 12/20/17 02:23, Priyanka Gugale wrote: > >> +1 >> Sometimes this context is required. We shouldn't change any default >> behaviour other than making this config available. >> >> -Priyanka >> >> >> >> On Wed, Dec 20, 2017 at 5:32 AM, Pramod Immaneni <pra...@datatorrent.com> >> wrote: >> >> The external system recording was just an example, not a specific use >>> case. >>> The idea is to provide comprehensive information to populateDAG as to the >>> context it is being called under. It is akin to the test mode or simulate >>> flag that you see with various utilities. The platform cannot control >>> what >>> populateDAG does, even without this information, in multiple calls that >>> you >>> mention the application can return different DAGs by depending on >>> any external factor such as time of day or some external variable. This >>> is >>> to merely provide more context information in the config. It is upto the >>> application to do what it wishes with it. >>> >>> On Tue, Dec 19, 2017 at 2:28 PM, Vlad Rozov <vro...@apache.org> wrote: >>> >>> -0.5: populateDAG() may be called by the platform as many times as it >>>> needs (even in case it calls it only once now to launch an application). >>>> Passing different parameters to populateDAG() in simulate launch mode >>>> and >>>> actual launch may lead to different DAG being constructed for those two >>>> modes. Can't the use case you described be handled by a plugin? >>>> >>>> Thank you, >>>> >>>> Vlad >>>> >>>> >>>> On 12/19/17 10:06, Sanjay Pujare wrote: >>>> >>>> +1 although I prefer something that is more enforceable. So I like the >>>>> idea >>>>> of another method but that introduces incompatibility so may be in 4.0? >>>>> >>>>> On Tue, Dec 19, 2017 at 9:40 AM, Munagala Ramanath < >>>>> amberar...@yahoo.com.invalid> wrote: >>>>> >>>>> +1 >>>>> >>>>>> Ram >>>>>> On Tuesday, December 19, 2017, 8:33:21 AM PST, Pramod Immaneni < >>>>>> pra...@datatorrent.com> wrote: >>>>>> >>>>>> I have a mini proposal. The command get-app-package-info runs the >>>>>> populateDAG method of an application to construct the DAG but does not >>>>>> actually launch the DAG. An application developer does not know in >>>>>> >>>>> which >>> >>>> context the populateDAG is being called. For example, if they are >>>>>> recording >>>>>> application starts in an external system from populateDAG, they will >>>>>> >>>>> have >>> >>>> false entries there. This can be solved in different ways such as >>>>>> introducing another method in StreamingApplication or more parameters >>>>>> to populateDAG but a non disruptive option would be to add a property >>>>>> >>>>> in >>> >>>> the configuration object that is passed to populateDAG to indicate if >>>>>> >>>>> it >>> >>>> is >>>>>> simulate/test mode or real launch. An application developer can use >>>>>> >>>>> this >>> >>>> property to take the appropriate actions. >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> >>>>>> >
