Re: Recommending or requiring mesos dns?

[John Omernik]

So I may be lookng at this wrong, but where is the data for the rm stored
if it does fail over? How will it know to pick up where it left off?  This
is just one area I am low in understanding on.

 That said, what about pre allocating a second failover rm some where on
the cluster.  (I am just tossing an idea here, in that there are probably
many reasons not to do this) but here is how I could see it happening.

1. Myriad starts a rm asking for 5 random available ports.  Mesos replies
starting the rm and reports to myriad the 5 ports used for the services you
listed below.

2. Myriad then checks a config value of number of "hot spares" lets say we
specify 1. Myriad then puts in a resource request to mesos for CPU and
memory required for the rm, but specifically asks for the same 5 ports
allocated to the first. Basically it reserves a spot on another node with
the same ports available. It may tak a bit, but there should be that
availability. Until this request is met, the yarn cluster is in a ha
compromised position.

3. At this point the perhaps we start another instance of rm right away
(depends on my first question on where the rm stores into about
jobs/applications) or the frame work just holds the spot, waiting for a
lack of heart beat (failover condition) on the primay resource manager.

4. If we can run the spare with no issues, it's a simple update of the dns
record and node managers connect to the new rm ( and another rm is
preallocated for redundancy). If we can't actually execute the secondary rm
until failover conditions, we can now execute the new rm, and the ports
will be the same.

This may seem kludgey at first, but done correctly, it may actually limit
the length of failover time as the rm is preallocated.  Rms are not huge
from a resource perspective thus it may be a small cost for those who want
failover and multiple clusters (thus having dynamic ports)

I will keep thinking this through, and would welcome feedback.

On Thursday, May 7, 2015, Santosh Marella  wrote:

> Hi John,
>
>   Great views about extending mesos dns for rm's discovery. Some thoughts:
>1. There are 5 primary interfaces RM exposes that are bound to standard
> ports.
> a. RPC interface for clients that want to submit applications to
> YARN (port 8032).
> b. RPC interface for NMs to connect back/HB to RM (port 8031).
> c. RPC interface for App Masters to connect back/HB to RM (port
> 8030).
> d. RPC interface for admin to interact with RM via CLI (port 8033).
> e. Web Interface for RM's UI (port 8088).
>2. When we launch RM using Marathon, it's probably better to mention in
> marathon's config that RM will use the above ports. This is because, if RM
> doesn't listens on random ports (as opposed to the above listed standard
> ports), when RM fails over, the new RM gets ports that might be different
> from the ones used by the old RM. This makes the RM's discovery hard,
> especially post failover.
>3. It looks like what you are proposing is a way to update mesos-dns as
> to what ports RM's services are listening on. And when RM fails over, these
> ports would get updated in mesos-dns. Is my understanding correct? If yes,
> one challenge I see is that the clients that want to connect to the above
> listed RM interfaces also need to pull the changes to RM's port numbers
> from mesos-dns dynamically. Not sure how that might be possible.
>
>   Regarding your question about NM ports
>   1. NM has the following ports:
>   a. RPC port for app masters to launch containers (this is a random
> port).
>   b. RPC port for localization service. (port 8040)
>   c. Web port for NM's UI (port 8042).
>2. Ports (a) and (c) are relayed to RM when NM registers with RM. Port
> (b) is passed to a local container executor process via command line args.
>3. As you rightly reckon, we need a mechanism at launch of NM to pass
> the mesos allocated ports to NM for the above interfaces. We can try
> to use variable
> expansion
> <
> http://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/conf/Configuration.html
> >
> mechanism hadoop has to achieve this.
>
> Thanks,
> Santosh
>
> On Thu, May 7, 2015 at 3:51 AM, John Omernik  > wrote:
>
> > I've implemented mesos-dns and use marathon to launch my myriad
> framework.
> > It shows up as myriad.marahon.mesos and makes it easy to find what node
> the
> > framework launched the resource manager on.
> >
> >  What if we made myriad mesos-dns aware, and prior to launching the yarn
> > rm, it could register in mesos dns. This would mean both the ip addresses
> > and the ports (we need to figure out multiple ports in mesos-dns). Then
> it
> > could write out ports and host names in the nm configs by checking mesos
> > dns for which ports the resource manager is using.
>
>
> > Side question:  when a node manager registers with the resource manager
> > are the ports the nm is running on completely up to the nm? Ie I can run
> my
> > nm web 

Re: Recommending or requiring mesos dns?

[Santosh Marella]

Hi John,

  Great views about extending mesos dns for rm's discovery. Some thoughts:
   1. There are 5 primary interfaces RM exposes that are bound to standard
ports.
a. RPC interface for clients that want to submit applications to
YARN (port 8032).
b. RPC interface for NMs to connect back/HB to RM (port 8031).
c. RPC interface for App Masters to connect back/HB to RM (port
8030).
d. RPC interface for admin to interact with RM via CLI (port 8033).
e. Web Interface for RM's UI (port 8088).
   2. When we launch RM using Marathon, it's probably better to mention in
marathon's config that RM will use the above ports. This is because, if RM
doesn't listens on random ports (as opposed to the above listed standard
ports), when RM fails over, the new RM gets ports that might be different
from the ones used by the old RM. This makes the RM's discovery hard,
especially post failover.
   3. It looks like what you are proposing is a way to update mesos-dns as
to what ports RM's services are listening on. And when RM fails over, these
ports would get updated in mesos-dns. Is my understanding correct? If yes,
one challenge I see is that the clients that want to connect to the above
listed RM interfaces also need to pull the changes to RM's port numbers
from mesos-dns dynamically. Not sure how that might be possible.

  Regarding your question about NM ports
  1. NM has the following ports:
  a. RPC port for app masters to launch containers (this is a random
port).
  b. RPC port for localization service. (port 8040)
  c. Web port for NM's UI (port 8042).
   2. Ports (a) and (c) are relayed to RM when NM registers with RM. Port
(b) is passed to a local container executor process via command line args.
   3. As you rightly reckon, we need a mechanism at launch of NM to pass
the mesos allocated ports to NM for the above interfaces. We can try
to use variable
expansion

mechanism hadoop has to achieve this.

Thanks,
Santosh

On Thu, May 7, 2015 at 3:51 AM, John Omernik  wrote:

> I've implemented mesos-dns and use marathon to launch my myriad framework.
> It shows up as myriad.marahon.mesos and makes it easy to find what node the
> framework launched the resource manager on.
>
>  What if we made myriad mesos-dns aware, and prior to launching the yarn
> rm, it could register in mesos dns. This would mean both the ip addresses
> and the ports (we need to figure out multiple ports in mesos-dns). Then it
> could write out ports and host names in the nm configs by checking mesos
> dns for which ports the resource manager is using.


> Side question:  when a node manager registers with the resource manager
> are the ports the nm is running on completely up to the nm? Ie I can run my
> nm web server any port, Yarn just explains that to the rm on registration?
> Because then we need a mechanism at launch of the nm task to understand
> which ports mesos has allocated to the nm and update the yarn-site for that
> nm before launch Perhaps mesos-dns as a requirement isn't needed, but I
> am trying to walk through options that get us closer to multiple yarn
> clusters on a mesos cluster.
>
> John
>
>
> --
> Sent from my iThing
>

Re: dev Digest of: get.107_109

[Justin Mclean]

Hi,

> 1. we do have a hangout to discuss the project every 2 weeks.   Is this not 
> appropriate?

There’s nothing wrong with that and if fact I would assume that helps build 
community. It would certailly help if the minutes/summary are posted to the 
mailing list, in plain text preferably, posting to a doc may mean that 
eventually that information is lost and preferably in the same thread. That way 
people new to the project or those not involved in the meetup can look at the 
email archives and get a better idea to what is going on. Any decisions however 
need to made on the mailing list. [1]

Thanks,
Justin

1. http://www.apache.org/dev/pmc.html#mailing-list-naming-policy

Re: Recommending or requiring mesos dns?

[John Omernik]

On point three, I have been running the rm with marathon which does put it
mesos - dns, I'd highly recommend this approach (thanks Santosh)
On May 7, 2015 8:37 AM, "Ken Sipe"  wrote:

> John,
>
> 1. +1 to mesos-dns aware
> 2. all tasks deployed by mesos are already in mesos-dns.  so all the nm
> are there (we should make sure they have good names.
> 3. the RM is not usually started with mesos… if it was it would also be
> listed in mesos-dns, however a process started outside mesos is not
> currently added to mesos-dns.  At some point mesos-dns will allow for out
> of band server registration… but it isn’t there today.
> 4. I would like to see multi-yarn clusters on mesos supported with
> multi-myriad.  Each myriad would managed it’s cluster and would register
> with a unique framework id.
>
> ken
>
> > On May 7, 2015, at 5:51 AM, John Omernik  wrote:
> >
> > I've implemented mesos-dns and use marathon to launch my myriad
> framework.
> > It shows up as myriad.marahon.mesos and makes it easy to find what node
> the
> > framework launched the resource manager on.
> >
> > What if we made myriad mesos-dns aware, and prior to launching the yarn
> > rm, it could register in mesos dns. This would mean both the ip addresses
> > and the ports (we need to figure out multiple ports in mesos-dns). Then
> it
> > could write out ports and host names in the nm configs by checking mesos
> > dns for which ports the resource manager is using.
> >
> > Side question:  when a node manager registers with the resource manager
> > are the ports the nm is running on completely up to the nm? Ie I can run
> my
> > nm web server any port, Yarn just explains that to the rm on
> registration?
> > Because then we need a mechanism at launch of the nm task to understand
> > which ports mesos has allocated to the nm and update the yarn-site for
> that
> > nm before launch Perhaps mesos-dns as a requirement isn't needed,
> but I
> > am trying to walk through options that get us closer to multiple yarn
> > clusters on a mesos cluster.
> >
> > John
> >
> >
> > --
> > Sent from my iThing
>
>

Re: Recommending or requiring mesos dns?

[Ken Sipe]

John,

1. +1 to mesos-dns aware
2. all tasks deployed by mesos are already in mesos-dns.  so all the nm are 
there (we should make sure they have good names.
3. the RM is not usually started with mesos… if it was it would also be listed 
in mesos-dns, however a process started outside mesos is not currently added to 
mesos-dns.  At some point mesos-dns will allow for out of band server 
registration… but it isn’t there today.
4. I would like to see multi-yarn clusters on mesos supported with 
multi-myriad.  Each myriad would managed it’s cluster and would register with a 
unique framework id.

ken

> On May 7, 2015, at 5:51 AM, John Omernik  wrote:
> 
> I've implemented mesos-dns and use marathon to launch my myriad framework.
> It shows up as myriad.marahon.mesos and makes it easy to find what node the
> framework launched the resource manager on.
> 
> What if we made myriad mesos-dns aware, and prior to launching the yarn
> rm, it could register in mesos dns. This would mean both the ip addresses
> and the ports (we need to figure out multiple ports in mesos-dns). Then it
> could write out ports and host names in the nm configs by checking mesos
> dns for which ports the resource manager is using.
> 
> Side question:  when a node manager registers with the resource manager
> are the ports the nm is running on completely up to the nm? Ie I can run my
> nm web server any port, Yarn just explains that to the rm on registration?
> Because then we need a mechanism at launch of the nm task to understand
> which ports mesos has allocated to the nm and update the yarn-site for that
> nm before launch Perhaps mesos-dns as a requirement isn't needed, but I
> am trying to walk through options that get us closer to multiple yarn
> clusters on a mesos cluster.
> 
> John
> 
> 
> -- 
> Sent from my iThing

Recommending or requiring mesos dns?

[John Omernik]

I've implemented mesos-dns and use marathon to launch my myriad framework.
It shows up as myriad.marahon.mesos and makes it easy to find what node the
framework launched the resource manager on.

 What if we made myriad mesos-dns aware, and prior to launching the yarn
rm, it could register in mesos dns. This would mean both the ip addresses
and the ports (we need to figure out multiple ports in mesos-dns). Then it
could write out ports and host names in the nm configs by checking mesos
dns for which ports the resource manager is using.

Side question:  when a node manager registers with the resource manager
are the ports the nm is running on completely up to the nm? Ie I can run my
nm web server any port, Yarn just explains that to the rm on registration?
Because then we need a mechanism at launch of the nm task to understand
which ports mesos has allocated to the nm and update the yarn-site for that
nm before launch Perhaps mesos-dns as a requirement isn't needed, but I
am trying to walk through options that get us closer to multiple yarn
clusters on a mesos cluster.

John


-- 
Sent from my iThing

Re: May board report

[Adam Bordelon]

Thanks Justin,

1. As Ken mentioned, we have a regularly scheduled community-wide hangout
meeting that the entire dev@ mailing list is invited

to. I know that Apache says that if it isn't in email, it didn't happen, so
we also share

the meeting minutes in a publicly editable google doc. We've even had a
community member from China dial in (we're rescheduling to a little earlier
for his sake). We find it extremely helpful to get on a live call to
discuss updates, issues, and action items. Is there anything wrong with
that? I can think of a few things we can do to raise
awareness/visibility/transparency.
a. Try to record the hangouts (because notes don't capture everything)
b. Be more vocal announcing the meetings before and sharing the minutes
afterwards.
c. Start an (archived) IRC channel for round-the-clock real-time
interaction.
d. Schedule hangouts at multiple times, to support people in other time
zones.

2. We must wait for the ebay code donation before migrating. We were also
waiting for committer ICLAs to be submitted, so that we could give all the
initial committers the karma to actually commit to the Apache repo, but all
of those ICLAs have now been received. Next steps are to work on the
ip-clearance form and freeze the codebase to submit a git sha snapshot.

3. The Docker images are only supplementary to a "release" of Myriad, and
are not required for installing/using it. Official releases will be tagged
git shas with all the hoopla surrounding the announcement. We still haven't

BTW, I'm filling out the Monthly Report now. Should be ready for review
soon.

On Wed, May 6, 2015 at 11:57 PM, Ken Sipe  wrote:

> Justin,
>
> Thanks for the detailed feedback.   As someone involved with the project
> here is my understanding:
>
> 1. we do have a hangout to discuss the project every 2 weeks.   Is this
> not appropriate?It really helps to sync up with where we are in the
> process. Would it help if we had meeting minutes and posted that the
> email list?
>
> 2.  most of the work and PR are in a non ASF git repo at the moment.  I
> believe we were waiting for eBay to officially donate the code (tied up in
> legal last I remember).   At the time the donation is official we were
> going to change the package namespace and move to the ASF repo.
>
> 3. I would need to review in more detail.
>
> We really appreciate your involvement and feedback!  Looking forward to
> more clarification if you have thoughts on my questions.
>
> ken
> > On May 6, 2015, at 11:56 PM, Justin Mclean 
> wrote:
> >
> > Hi,
> >
> > I’ve been appointed shepherd for Myriad this month and while doing a
> review  I noticed a couple of minor things that you may wish to address in
> your board report and/or on this dev list.
> >
> > Please note that this is just from a casual glance and you certainly
> know your project better that I do and I may of missed something/there may
> not be any issues at all.
> >
> > 1. Conversations seems to be happening off list [1].
> > 2. Git report has been created but there’s no code in it [2]. Pull
> requests are still happening on non ASF github.
> > 3. May need some clarity around making releases in this way [3] I
> suggest you carefully read [4] [5]. Note that "Under no circumstances are
> unapproved builds a substitute for releases."
> >
> > Again I’m not involved in this project and it up to the PMC (with the
> Mentors help) to decide what to do and don’t do, so feel free to ignore the
> above if you don’t think there’s any issue here.
> >
> > Thanks,
> > Justin
> >
> > 1.
> http://mail-archives.apache.org/mod_mbox/incubator-myriad-dev/201504.mbox/%3ccakoqndusesuoko4yraqwsjblvjrhstbcqghcngwynuhzym7...@mail.gmail.com%3e
> > 2.
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=summary
> > 3.
> http://mail-archives.apache.org/mod_mbox/incubator-myriad-dev/201504.mbox/%3c985b60f3-eb09-4b54-8b4a-6a791c151...@mesosphere.io%3e
> > 4.
> http://www.apache.org/dev/release.html#what-must-every-release-contain
> > 5. http://www.apache.org/dev/release.html#what
>
>