+1. I like the idea.
For me, submarine/ML-job-execution seems to be a natural extension of the existing Hadoop/Yarn capabilities. And like the proposed project structure / release lifecycle, too. I think it's better to be more modularized but keep the development in the same project. IMHO it worked well with the Ozone releases. We can do more frequent releases and support multiple versions of core hadoop but the tested new improvements could be moved back to the hadoop-common. Marton On 1/31/19 7:53 PM, Wangda Tan wrote: > Hi devs, > > Since we started submarine-related effort last year, we received a lot of > feedbacks, several companies (such as Netease, China Mobile, etc.) are > trying to deploy Submarine to their Hadoop cluster along with big data > workloads. Linkedin also has big interests to contribute a Submarine TonY ( > https://github.com/linkedin/TonY) runtime to allow users to use the same > interface. > > From what I can see, there're several issues of putting Submarine under > yarn-applications directory and have same release cycle with Hadoop: > > 1) We started 3.2.0 release at Sep 2018, but the release is done at Jan > 2019. Because of non-predictable blockers and security issues, it got > delayed a lot. We need to iterate submarine fast at this point. > > 2) We also see a lot of requirements to use Submarine on older Hadoop > releases such as 2.x. Many companies may not upgrade Hadoop to 3.x in a > short time, but the requirement to run deep learning is urgent to them. We > should decouple Submarine from Hadoop version. > > And why we wanna to keep it within Hadoop? First, Submarine included some > innovation parts such as enhancements of user experiences for YARN > services/containerization support which we can add it back to Hadoop later > to address common requirements. In addition to that, we have a big overlap > in the community developing and using it. > > There're several proposals we have went through during Ozone merge to trunk > discussion: > https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201803.mbox/%3ccahfhakh6_m3yldf5a2kq8+w-5fbvx5ahfgs-x1vajw8gmnz...@mail.gmail.com%3E > > I propose to adopt Ozone model: which is the same master branch, different > release cycle, and different release branch. It is a great example to show > agile release we can do (2 Ozone releases after Oct 2018) with less > overhead to setup CI, projects, etc. > > *Links:* > - JIRA: https://issues.apache.org/jira/browse/YARN-8135 > - Design doc > <https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit> > - User doc > <https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/Index.html> > (3.2.0 > release) > - Blogposts, {Submarine} : Running deep learning workloads on Apache Hadoop > <https://hortonworks.com/blog/submarine-running-deep-learning-workloads-apache-hadoop/>, > (Chinese Translation: Link <https://www.jishuwen.com/d/2Vpu>) > - Talks: Strata Data Conf NY > <https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/68289> > > Thoughts? > > Thanks, > Wangda Tan > --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
