Re: 0.2.0 release
Swanil, I concur and want to keep both options for Mesos and Docker networking available, and putting the configuration for both in should be a priority. However, one has to be careful with this as the NM's register with the RM via heartbeats with their container port (Not the host port), this isn't an issue if NM and RM are in the same Docker Network, via Weave or Kubernetes but is with simple bridged networking. We also have to be careful as Myriad currently doesn't run HDFS itself so we'd lose data locality. My idea was the start with Host Networking so we could make Myriad easier to deploy but leave room to add additional networking options: basically exposing all the protobuf options for Docker Parameters (used to configure docker networking) and NetworkInfo (used to configure Mesos networking). Darin On Tue, Mar 22, 2016 at 2:48 PM, Swapnil Daingadewrote: > Hi Darin, > > I feel docker networking is something we should spent time to think > through. > A user should be able to use multiple options provided by Mesos, Docker, > 3rd party etc > > It would be great if we can abstract the specific implementation to provide > container ip addresses behind interfaces. User should be able to switch > implementations by making simple changes in configuration files. > > Regards > Swapnil > > > On Tue, Mar 22, 2016 at 8:20 AM, Darin Johnson > wrote: > > > Swapnil, > > > > Any help would be appreciated. I'll try to write up what I'm working on > > tomorrow. But essentially the ideas are: > > 1. Ability to launch the resource manager and node managers in docker > > containers > > 2. Use host networking for now (Ports configured to be pulled from mesos > - > > ability to use ports reserved by role), but leave hooks to easily add IP > > per container. > > 3. Ability to get configuration files for a URI > > 4. Ability to mount local volumes for local directories in the shuffle > > phase etc (though will require more config). > > > > Darin > > >
Re: 0.2.0 release
Hi Darin, I feel docker networking is something we should spent time to think through. A user should be able to use multiple options provided by Mesos, Docker, 3rd party etc It would be great if we can abstract the specific implementation to provide container ip addresses behind interfaces. User should be able to switch implementations by making simple changes in configuration files. Regards Swapnil On Tue, Mar 22, 2016 at 8:20 AM, Darin Johnsonwrote: > Swapnil, > > Any help would be appreciated. I'll try to write up what I'm working on > tomorrow. But essentially the ideas are: > 1. Ability to launch the resource manager and node managers in docker > containers > 2. Use host networking for now (Ports configured to be pulled from mesos - > ability to use ports reserved by role), but leave hooks to easily add IP > per container. > 3. Ability to get configuration files for a URI > 4. Ability to mount local volumes for local directories in the shuffle > phase etc (though will require more config). > > Darin >
Re: 0.2.0 release
Swapnil, Any help would be appreciated. I'll try to write up what I'm working on tomorrow. But essentially the ideas are: 1. Ability to launch the resource manager and node managers in docker containers 2. Use host networking for now (Ports configured to be pulled from mesos - ability to use ports reserved by role), but leave hooks to easily add IP per container. 3. Ability to get configuration files for a URI 4. Ability to mount local volumes for local directories in the shuffle phase etc (though will require more config). Darin
Re: 0.2.0 release
+1 on Darin as release manager I'd like to see 0.2 have: - Usable FGS - Dockerized NM (for multitenancy) On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson <dbjohnson1...@gmail.com> wrote: > We've talked about a 0.2.0 release slated for mid April at the dev sync. > I'd like to nail down any features people would like and have time to work > on. > > I've been spend some time fixing major bugs to the FGS feature and plan to > address MYRIAD-136 and MYRIAD-189. > > I'd also be willing to be the release manager on this release if necessary. > > Darin >
Re: 0.2.0 release
+1 Darin for RM. Would like be involved in the multi-tenency with Dockerized NM work. Perhaps we can start filing JIRA's targeted for 0.2 and start adding design proposals for people to review and comment. Regards Swapnil On Thu, Mar 17, 2016 at 8:13 AM, Darin Johnson <dbjohnson1...@gmail.com> wrote: > Happy to report as of the last two PRS, FGS is usable no memory leaks or > crashes, could likely be improved with fancier schedulers but that's for > the future. I'm currently looking at running some terasort benchmarks with > FGS and a reserved resources vs statically sized NMs to figure out the > performance hit. Might be worth a blog post in the near future. > > Adam, I've been looking through the cgroups code for myriad recently, > apparently we need the mod the path YARN uses for it's Hierarchy. Does > that change at all within a Docker container or is it the same? > > Darin > > On Mar 16, 2016 8:48 PM, "Adam Bordelon" <a...@mesosphere.io> wrote: > > > +1 on Darin as release manager > > > > I'd like to see 0.2 have: > > - Usable FGS > > - Dockerized NM (for multitenancy) > > > > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson <dbjohnson1...@gmail.com> > > wrote: > > > > > We've talked about a 0.2.0 release slated for mid April at the dev > sync. > > > I'd like to nail down any features people would like and have time to > > work > > > on. > > > > > > I've been spend some time fixing major bugs to the FGS feature and plan > > to > > > address MYRIAD-136 and MYRIAD-189. > > > > > > I'd also be willing to be the release manager on this release if > > necessary. > > > > > > Darin > > > > > >
Re: 0.2.0 release
Adam: I agree with your suggestion for 0.2. I am, however, curious to understand why a dockerized NM would be a requirement for multi-tenancy. I don't disagree, I'm just trying to understand. paul On Wed, Mar 16, 2016 at 8:48 PM, Adam Bordelon <a...@mesosphere.io> wrote: > +1 on Darin as release manager > > I'd like to see 0.2 have: > - Usable FGS > - Dockerized NM (for multitenancy) > > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson <dbjohnson1...@gmail.com> > wrote: > > > We've talked about a 0.2.0 release slated for mid April at the dev sync. > > I'd like to nail down any features people would like and have time to > work > > on. > > > > I've been spend some time fixing major bugs to the FGS feature and plan > to > > address MYRIAD-136 and MYRIAD-189. > > > > I'd also be willing to be the release manager on this release if > necessary. > > > > Darin > > > -- *Paul Curtis *- Senior Product Technologist *O: *+1 203-660-0015 - *M:* +1 203-539-9705 Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email_medium=Signature_campaign=Free%20available>
Re: 0.2.0 release
Happy to report as of the last two PRS, FGS is usable no memory leaks or crashes, could likely be improved with fancier schedulers but that's for the future. I'm currently looking at running some terasort benchmarks with FGS and a reserved resources vs statically sized NMs to figure out the performance hit. Might be worth a blog post in the near future. Adam, I've been looking through the cgroups code for myriad recently, apparently we need the mod the path YARN uses for it's Hierarchy. Does that change at all within a Docker container or is it the same? Darin On Mar 16, 2016 8:48 PM, "Adam Bordelon" <a...@mesosphere.io> wrote: > +1 on Darin as release manager > > I'd like to see 0.2 have: > - Usable FGS > - Dockerized NM (for multitenancy) > > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson <dbjohnson1...@gmail.com> > wrote: > > > We've talked about a 0.2.0 release slated for mid April at the dev sync. > > I'd like to nail down any features people would like and have time to > work > > on. > > > > I've been spend some time fixing major bugs to the FGS feature and plan > to > > address MYRIAD-136 and MYRIAD-189. > > > > I'd also be willing to be the release manager on this release if > necessary. > > > > Darin > > >
0.2.0 release
We've talked about a 0.2.0 release slated for mid April at the dev sync. I'd like to nail down any features people would like and have time to work on. I've been spend some time fixing major bugs to the FGS feature and plan to address MYRIAD-136 and MYRIAD-189. I'd also be willing to be the release manager on this release if necessary. Darin