Hi all, I realized the details in my previous mail were sparing. From my understanding of https://github.com/hadoopsubmarine/submarine-installer/blob/master/InstallationGuide.md - it seems to get Submarine working - two things are essential -
1. Docker Runtime in YARN 2. Some non-trivial Docker Networking setup / configuration so that Docker containers launched by submarine can seamlessly communicate Step 2 involves some non-trivial configuration so that etcd, calico, yarn-registry-dns-service can seamlessly interact. Is my understanding correct? I've setup vanilla hadoop before and it is easy. However, after trying submarine setup for two days I'm still struggling. As someone who is new to docker networking and yarn-dns-registry-service - please can someone point me to what are the minimal essential steps that I should follow to get Submarine working. I wish to contribute to Submarine. I'm a graduate student and my aim is to try different scheduling schemes (implemented in YARN's RM) and see how different ML training workloads (submitted using Submarine) perform. In doing this, I hope to make some contributions to Submarine - like perhaps developing the pause / resume feature in submarine. Any help will be much appreciated. Thanks, Kshiteej On Sat, Mar 23, 2019 at 8:35 PM Kshiteej Mahajan <[email protected]> wrote: > Hi all, > > I've been trying to follow the guide at > https://github.com/hadoopsubmarine/submarine-installer/blob/master/InstallationGuide.md > - > but there are a lot of pre-requisites like Calico, HBase, Zookeeper that > are needed with that guide. Are all these pre-reqs essential to a submarine > setup. > > Please can someone point me to a bare minimal installation guide for > setting up submarine. > > Thanks, > Kshiteej >
