Hi Jangho. Thanks for sharing the meeting result! I especially like the off-topic contributions about "Stackable" SchedulingPolicy. I have an implementation of intermediate data location aware scheduling policy, but it was hard to come up with a way to concisely merge this feature to our master. According to the issue description of the stackable scheduling policy, it seems that it will resolve my concern. Thanks!
Sincerely, Sanha 2018-05-10 15:49 GMT+09:00 JangHo Seo <[email protected]>: > Sorry for the bad formatting. I'm sharing the Google Docs link: > > https://docs.google.com/document/d/1JoVh4pvt89fZMHnvJNTgVKEabOzHp > 8HH0mHJ8rvVBwE/edit?usp=sharing > > Regards, > Jangho Seo > > > On 05/10/2018 03:41 PM, JangHo Seo wrote: > > To Nemo devs, > > > > Our GSoC student Jaehyeon, and two mentors (Wonook and I) had a kick-off > > meeting for GSoC activity today. > > And I'm here to share the minutes. > > > > ==== MEETING MINUTES ==== > > > > ## Feedbacks from Mentors on GSoC Proposal > > > > - Scheduling breakdown is well-organized and quite concrete. > > - 'Description of work' is somewhat abstract. > > - The concept of 'common interface on visualization for Nemo modules' > > seems very promising. > > > > ## Discussion on Work Description & Schedule > > > > The entire work can be divided into two big sides - Nemo runtime and web > > frontend. > > > > - Nemo Runtime (Main language: Java. Will be addressed before early > July.) > > - Use reef-webserver to provide JSON-formatted metric data from Nemo > > backbone and pluggable modules. > > - To collect metric data from modules, define and use common > > interface for visualization > > so that future module developers can easily add support for > > visualization. > > - Implement appropriate routing to support RESTful requests from web > > frontend. > > - Web Frontend (Main language: Javscript. Will be addressed after early > > July.) > > - Implement web frontend which polls states and metrics from Nemo > > backbone and display them. > > - Improve web frontend to visualize generic directed acyclic > graph(DAG). > > - Improve frontend to visualize metrics from Nemo modules. > > > > We decided to address Nemo runtime-related issues before touching web > > frontend, > > so that Jaehyeon can save time by not having to write mocks to test web > > frontend. > > > > And this is the revised schedule: > > > > May 14 - June 1 > > - Define RESTful API specification. > > - Provide basic status and statistics from Nemo backbone through JSON, > > such as job state, > > stage state, TaskGroup state, and job completion time. > > > > June 1 - June 15 (first evaluation deadline) > > - Implement RESTful API stack on Nemo. > > - Improve Nemo RESTful API implementation to provide DAG-based > > information, such as Nemo IRs > > after optimization passes, physical execution plans, and scheduling > > dependencies. > > > > June 15 - July 1 > > - Write interface for Nemo runtime modules to provide the internal > > states to the visualizer. > > - Modify some provided implementations of Nemo runtime modules to > > present internal states > > via the interface mentioned above. > > > > July 1- July 13 (second evaluation deadline) > > - Implement web frontend which polls states and metrics from Nemo > > runtime master and display them. > > - Improve web frontend to visualize generic directed acyclic graph(DAG). > > > > July 13 - August 1 > > - Improve web frontend to display information from Nemo runtime modules. > > - Re-read documents and make refinements. > > > > ## Making Early Small Contribution > > > > - Mentors suggested and Jaehyeon agreed to try 'off-topic' contributions > > on Nemo. > > - We picked '[NEMO-53] Make SchedulingPolicy Stackable'. > > Solving this issue is not extremely hard, but it will have good impact > > on the project. > > - For this issue, Jangho is responsible for the code review and > mentoring. > > > > ## Miscellaneous > > > > - Mentors checked and ensured that JaeHyeon has enough access to Nemo > > community infra. > > (ASF JIRA (username: skystar), Nemo Slack worksapce) > > > > ==== END OF MEETING MINUTES ==== > > > > As always, feedbacks welcome. > > > > Regards, > > Jangho Seo > > > >
