Re: 0.2.0 release

[Darin Johnson]

Swanil,

I concur and want to keep both options for Mesos and Docker networking
available, and putting the configuration for both in should be a priority.
However, one has to be careful with this as the NM's register with the RM
via heartbeats with their container port (Not the host port), this isn't an
issue if NM and RM are in the same Docker Network, via Weave or Kubernetes
but is with simple bridged networking. We also have to be careful as Myriad
currently doesn't run HDFS itself so we'd lose data locality.  My idea was
the start with Host Networking so we could make Myriad easier to deploy but
leave room to add additional networking options: basically exposing all the
protobuf options for Docker Parameters (used to configure docker
networking) and NetworkInfo (used to configure Mesos networking).

Darin

On Tue, Mar 22, 2016 at 2:48 PM, Swapnil Daingade 
wrote:

> Hi Darin,
>
> I feel docker networking is something we should spent time to think
> through.
> A user should be able to use multiple options provided by Mesos, Docker,
> 3rd party etc
>
> It would be great if we can abstract the specific implementation to provide
> container ip addresses behind interfaces. User should be able to switch
> implementations by making simple changes in configuration files.
>
> Regards
> Swapnil
>
>
> On Tue, Mar 22, 2016 at 8:20 AM, Darin Johnson 
> wrote:
>
> > Swapnil,
> >
> > Any help would be appreciated.  I'll try to write up what I'm working on
> > tomorrow.  But essentially the ideas are:
> > 1. Ability to launch the resource manager and node managers in docker
> > containers
> > 2. Use host networking for now (Ports configured to be pulled from mesos
> -
> > ability to use ports reserved by role), but leave hooks to easily add IP
> > per container.
> > 3. Ability to get configuration files for a URI
> > 4. Ability to mount local volumes for local directories in the shuffle
> > phase etc (though will require more config).
> >
> > Darin
> >
>

Re: 0.2.0 release

[Swapnil Daingade]

Hi Darin,

I feel docker networking is something we should spent time to think through.
A user should be able to use multiple options provided by Mesos, Docker,
3rd party etc

It would be great if we can abstract the specific implementation to provide
container ip addresses behind interfaces. User should be able to switch
implementations by making simple changes in configuration files.

Regards
Swapnil


On Tue, Mar 22, 2016 at 8:20 AM, Darin Johnson 
wrote:

> Swapnil,
>
> Any help would be appreciated.  I'll try to write up what I'm working on
> tomorrow.  But essentially the ideas are:
> 1. Ability to launch the resource manager and node managers in docker
> containers
> 2. Use host networking for now (Ports configured to be pulled from mesos -
> ability to use ports reserved by role), but leave hooks to easily add IP
> per container.
> 3. Ability to get configuration files for a URI
> 4. Ability to mount local volumes for local directories in the shuffle
> phase etc (though will require more config).
>
> Darin
>

Re: 0.2.0 release

[Darin Johnson]

Swapnil,

Any help would be appreciated.  I'll try to write up what I'm working on
tomorrow.  But essentially the ideas are:
1. Ability to launch the resource manager and node managers in docker
containers
2. Use host networking for now (Ports configured to be pulled from mesos -
ability to use ports reserved by role), but leave hooks to easily add IP
per container.
3. Ability to get configuration files for a URI
4. Ability to mount local volumes for local directories in the shuffle
phase etc (though will require more config).

Darin

Re: 0.2.0 release

[Adam Bordelon]

+1 on Darin as release manager

I'd like to see 0.2 have:
- Usable FGS
- Dockerized NM (for multitenancy)

On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson 
wrote:

> We've talked about a 0.2.0 release slated for mid April at the dev sync.
> I'd like to nail down any features people would like and have time to work
> on.
>
> I've been spend some time fixing major bugs to the FGS feature and plan to
> address MYRIAD-136 and MYRIAD-189.
>
> I'd also be willing to be the release manager on this release if necessary.
>
> Darin
>

Re: 0.2.0 release

[Swapnil Daingade]

+1 Darin for RM.

Would like be involved in the multi-tenency with Dockerized NM work.
Perhaps we can start filing JIRA's targeted for 0.2 and start adding design
proposals
for people to review and comment.

Regards
Swapnil


On Thu, Mar 17, 2016 at 8:13 AM, Darin Johnson 
wrote:

> Happy to report as of the last two PRS, FGS is usable no memory leaks or
> crashes, could likely be improved with fancier schedulers but that's for
> the future.  I'm currently looking at running some terasort benchmarks with
> FGS and a reserved resources vs statically sized NMs to figure out the
> performance hit.  Might be worth a blog post in the near future.
>
> Adam, I've been looking through the cgroups code for myriad recently,
> apparently we need the mod the path YARN uses for it's Hierarchy.  Does
> that change at all within a Docker container or is it the same?
>
> Darin
>
> On Mar 16, 2016 8:48 PM, "Adam Bordelon"  wrote:
>
> > +1 on Darin as release manager
> >
> > I'd like to see 0.2 have:
> > - Usable FGS
> > - Dockerized NM (for multitenancy)
> >
> > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson 
> > wrote:
> >
> > > We've talked about a 0.2.0 release slated for mid April at the dev
> sync.
> > > I'd like to nail down any features people would like and have time to
> > work
> > > on.
> > >
> > > I've been spend some time fixing major bugs to the FGS feature and plan
> > to
> > > address MYRIAD-136 and MYRIAD-189.
> > >
> > > I'd also be willing to be the release manager on this release if
> > necessary.
> > >
> > > Darin
> > >
> >
>

Re: 0.2.0 release

[Paul Curtis]

Adam:

I agree with your suggestion for 0.2. I am, however, curious to understand
why a dockerized NM would be a requirement for multi-tenancy. I don't
disagree, I'm just trying to understand.

paul

On Wed, Mar 16, 2016 at 8:48 PM, Adam Bordelon  wrote:

> +1 on Darin as release manager
>
> I'd like to see 0.2 have:
> - Usable FGS
> - Dockerized NM (for multitenancy)
>
> On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson 
> wrote:
>
> > We've talked about a 0.2.0 release slated for mid April at the dev sync.
> > I'd like to nail down any features people would like and have time to
> work
> > on.
> >
> > I've been spend some time fixing major bugs to the FGS feature and plan
> to
> > address MYRIAD-136 and MYRIAD-189.
> >
> > I'd also be willing to be the release manager on this release if
> necessary.
> >
> > Darin
> >
>



-- 
*Paul Curtis *- Senior Product Technologist
*O: *+1 203-660-0015 - *M:* +1 203-539-9705

Now Available - Free Hadoop On-Demand Training

Re: 0.2.0 release

[Darin Johnson]

Happy to report as of the last two PRS, FGS is usable no memory leaks or
crashes, could likely be improved with fancier schedulers but that's for
the future.  I'm currently looking at running some terasort benchmarks with
FGS and a reserved resources vs statically sized NMs to figure out the
performance hit.  Might be worth a blog post in the near future.

Adam, I've been looking through the cgroups code for myriad recently,
apparently we need the mod the path YARN uses for it's Hierarchy.  Does
that change at all within a Docker container or is it the same?

Darin

On Mar 16, 2016 8:48 PM, "Adam Bordelon"  wrote:

> +1 on Darin as release manager
>
> I'd like to see 0.2 have:
> - Usable FGS
> - Dockerized NM (for multitenancy)
>
> On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson 
> wrote:
>
> > We've talked about a 0.2.0 release slated for mid April at the dev sync.
> > I'd like to nail down any features people would like and have time to
> work
> > on.
> >
> > I've been spend some time fixing major bugs to the FGS feature and plan
> to
> > address MYRIAD-136 and MYRIAD-189.
> >
> > I'd also be willing to be the release manager on this release if
> necessary.
> >
> > Darin
> >
>