Hi Sanha, Thanks for your comment! Can't wait to see the intermediate data location aware scheduling merged into the master!
Regards, Jangho On 05/10/2018 04:08 PM, Sanha Lee wrote: > Hi Jangho. > Thanks for sharing the meeting result! > > I especially like the off-topic contributions about "Stackable" > SchedulingPolicy. > I have an implementation of intermediate data location aware scheduling > policy, but it was hard to come up with a way to concisely merge this > feature to our master. > According to the issue description of the stackable scheduling policy, it > seems that it will resolve my concern. > Thanks! > > Sincerely, > Sanha > > 2018-05-10 15:49 GMT+09:00 JangHo Seo <[email protected]>: > >> Sorry for the bad formatting. I'm sharing the Google Docs link: >> >> https://docs.google.com/document/d/1JoVh4pvt89fZMHnvJNTgVKEabOzHp >> 8HH0mHJ8rvVBwE/edit?usp=sharing >> >> Regards, >> Jangho Seo >> >> >> On 05/10/2018 03:41 PM, JangHo Seo wrote: >>> To Nemo devs, >>> >>> Our GSoC student Jaehyeon, and two mentors (Wonook and I) had a kick-off >>> meeting for GSoC activity today. >>> And I'm here to share the minutes. >>> >>> ==== MEETING MINUTES ==== >>> >>> ## Feedbacks from Mentors on GSoC Proposal >>> >>> - Scheduling breakdown is well-organized and quite concrete. >>> - 'Description of work' is somewhat abstract. >>> - The concept of 'common interface on visualization for Nemo modules' >>> seems very promising. >>> >>> ## Discussion on Work Description & Schedule >>> >>> The entire work can be divided into two big sides - Nemo runtime and web >>> frontend. >>> >>> - Nemo Runtime (Main language: Java. Will be addressed before early >> July.) >>> - Use reef-webserver to provide JSON-formatted metric data from Nemo >>> backbone and pluggable modules. >>> - To collect metric data from modules, define and use common >>> interface for visualization >>> so that future module developers can easily add support for >>> visualization. >>> - Implement appropriate routing to support RESTful requests from web >>> frontend. >>> - Web Frontend (Main language: Javscript. Will be addressed after early >>> July.) >>> - Implement web frontend which polls states and metrics from Nemo >>> backbone and display them. >>> - Improve web frontend to visualize generic directed acyclic >> graph(DAG). >>> - Improve frontend to visualize metrics from Nemo modules. >>> >>> We decided to address Nemo runtime-related issues before touching web >>> frontend, >>> so that Jaehyeon can save time by not having to write mocks to test web >>> frontend. >>> >>> And this is the revised schedule: >>> >>> May 14 - June 1 >>> - Define RESTful API specification. >>> - Provide basic status and statistics from Nemo backbone through JSON, >>> such as job state, >>> stage state, TaskGroup state, and job completion time. >>> >>> June 1 - June 15 (first evaluation deadline) >>> - Implement RESTful API stack on Nemo. >>> - Improve Nemo RESTful API implementation to provide DAG-based >>> information, such as Nemo IRs >>> after optimization passes, physical execution plans, and scheduling >>> dependencies. >>> >>> June 15 - July 1 >>> - Write interface for Nemo runtime modules to provide the internal >>> states to the visualizer. >>> - Modify some provided implementations of Nemo runtime modules to >>> present internal states >>> via the interface mentioned above. >>> >>> July 1- July 13 (second evaluation deadline) >>> - Implement web frontend which polls states and metrics from Nemo >>> runtime master and display them. >>> - Improve web frontend to visualize generic directed acyclic graph(DAG). >>> >>> July 13 - August 1 >>> - Improve web frontend to display information from Nemo runtime modules. >>> - Re-read documents and make refinements. >>> >>> ## Making Early Small Contribution >>> >>> - Mentors suggested and Jaehyeon agreed to try 'off-topic' contributions >>> on Nemo. >>> - We picked '[NEMO-53] Make SchedulingPolicy Stackable'. >>> Solving this issue is not extremely hard, but it will have good impact >>> on the project. >>> - For this issue, Jangho is responsible for the code review and >> mentoring. >>> ## Miscellaneous >>> >>> - Mentors checked and ensured that JaeHyeon has enough access to Nemo >>> community infra. >>> (ASF JIRA (username: skystar), Nemo Slack worksapce) >>> >>> ==== END OF MEETING MINUTES ==== >>> >>> As always, feedbacks welcome. >>> >>> Regards, >>> Jangho Seo >>> >>
