Sounds good to me.
I agree that policy-related code should go into 'nemo-common' for the compiler-optimizer to optimize, and the runtime to execute. I guess 'KeyExtractor' and the ir package belong to this category, and that's why they're in 'nemo-common' On the other hand, mechanism-related code should stay in 'nemo-runtime-common' and should be exposed only to the compiler-backend/nemo. Things like 'PhysicalStage' seems to belong to this category, as it'd be unlikely that compiler-optimizer would want to do something with it. Thanks, John On Fri, Jun 1, 2018 at 3:59 PM, JangHo Seo <[email protected]> wrote: > Hi Nemo devs, > > Currently Nemo codebase is splitted into multiple Maven modules. > While it's good for ease of maintenance and modularity, a module cannot > reference classes and/or interfaces in other modules unless it declares > them as dependencies in pom.xml. > Sometimes it makes difficult to expose runtime-level behaviors as > IR-level abstractions, when nemo-compiler-* modules cannot access the > related definitions in nemo-runtime-* modules. > > A good example is [NEMO-73] 'SchedulingPolicy as Vertex-levelExecution > Property'. The issue is to expose SchedulingPolicy to compiler passes so > that users can configure scheduling behavior using Nemo IR. > Unfortunately 'SchedulingPolicy' is defined in 'nemo-runtime-common', so > compilers cannot access it and I think the proper fix is to move it to > the 'nemo-common' module. > Also, to move 'SchedulingPolicy' to 'nemo-common' module, lots of > definitions in 'nemo-runtime-common' should also be moved to 'nemo-common'. > (Because 'SchedulingPolicy' is a function of 'ExecutorRepresenter's and > 'ExecutableTask', and the definition of 'ExecutableTask' depends on > 'PhysicalStageEdge', 'PhysicalStage', and 'KeyRange', and so on...) > > The module 'nemo-runtime-common' has definitions that define 'policies', > as well as definitions that implement 'mechanisms'. > 'KeyRange' is an example for the former (defines how to partition the > data in a block), and 'MessageListener' is an example for the latter > (implements RPC). > In my opinion, policy-related definitions should be moved to > nemo-common, so that we can add more ExecutionProperties about > scheduilng, dynamic optimization, state management, and so on. > > What do you think about this issue? > > Regards, > Jangho > >
