Hi John,
Great views about extending mesos dns for rm's discovery. Some thoughts:
1. There are 5 primary interfaces RM exposes that are bound to standard
ports.
a. RPC interface for clients that want to submit applications to
YARN (port 8032).
b. RPC interface for NMs to connect back/HB to RM (port 8031).
c. RPC interface for App Masters to connect back/HB to RM (port
8030).
d. RPC interface for admin to interact with RM via CLI (port 8033).
e. Web Interface for RM's UI (port 8088).
2. When we launch RM using Marathon, it's probably better to mention in
marathon's config that RM will use the above ports. This is because, if RM
doesn't listens on random ports (as opposed to the above listed standard
ports), when RM fails over, the new RM gets ports that might be different
from the ones used by the old RM. This makes the RM's discovery hard,
especially post failover.
3. It looks like what you are proposing is a way to update mesos-dns as
to what ports RM's services are listening on. And when RM fails over, these
ports would get updated in mesos-dns. Is my understanding correct? If yes,
one challenge I see is that the clients that want to connect to the above
listed RM interfaces also need to pull the changes to RM's port numbers
from mesos-dns dynamically. Not sure how that might be possible.
Regarding your question about NM ports
1. NM has the following ports:
a. RPC port for app masters to launch containers (this is a random
port).
b. RPC port for localization service. (port 8040)
c. Web port for NM's UI (port 8042).
2. Ports (a) and (c) are relayed to RM when NM registers with RM. Port
(b) is passed to a local container executor process via command line args.
3. As you rightly reckon, we need a mechanism at launch of NM to pass
the mesos allocated ports to NM for the above interfaces. We can try
to use variable
expansion
<http://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/conf/Configuration.html>
mechanism hadoop has to achieve this.
Thanks,
Santosh
On Thu, May 7, 2015 at 3:51 AM, John Omernik <[email protected]> wrote:
> I've implemented mesos-dns and use marathon to launch my myriad framework.
> It shows up as myriad.marahon.mesos and makes it easy to find what node the
> framework launched the resource manager on.
>
> What if we made myriad mesos-dns aware, and prior to launching the yarn
> rm, it could register in mesos dns. This would mean both the ip addresses
> and the ports (we need to figure out multiple ports in mesos-dns). Then it
> could write out ports and host names in the nm configs by checking mesos
> dns for which ports the resource manager is using.
> Side question: when a node manager registers with the resource manager
> are the ports the nm is running on completely up to the nm? Ie I can run my
> nm web server any port, Yarn just explains that to the rm on registration?
> Because then we need a mechanism at launch of the nm task to understand
> which ports mesos has allocated to the nm and update the yarn-site for that
> nm before launch.... Perhaps mesos-dns as a requirement isn't needed, but I
> am trying to walk through options that get us closer to multiple yarn
> clusters on a mesos cluster.
>
> John
>
>
> --
> Sent from my iThing
>
