OK
Thank you for your feedback.
This approach does not prohibit you from continuing running JHS, TLS, others
from Marathon(s) as you do it today. It is in addition and not a replacement.
I would like to hear from more people before considering this feature a no-op.
Thanks,Yuliya
From: Darin Johnson <[email protected]>
To: [email protected]; yuliya Feldman <[email protected]>
Sent: Thursday, July 23, 2015 9:19 AM
Subject: Re: JobHistoryServer running as MyriadTask
I was planning to run the resource manager (hopefully two via marathon once
ha is done) and the history server via marathon. The three cons listed
don't seem too severe. Addressing the cons:
1.
In case of multiple Myriad clusters and single Marathon instance
services will rely on FIFO within Marathon to get resources
Not as bad as you'd think, FIFO is pretty safe for long running
resources. You're not going to be hit that hard. Added benefit, the TLS,
JHS, RM are running on a Marathon framework with a potentially better role.
I do this already for one instance of Marathon for services I want to give
high priority to. In general, I'd be inclined to leave a NM to a lower
priority role. I want my M/R jobs to eventually run, I want the TLS to be
running not eventually running.
2.
Some env. variables, settings that Myriad already has like RM host/ports
may not be easily obtainable/available from outside of Myriad framework(s).
If running the RM in Marathon the relevant configs should be obtainable
via a URI already. I'm caveating the on the fact I already rely on a DNS
style service discovery (Mesos-DNS and Consul-Mesos both work fine for
this).
3. When tearing down Myriad cluster those services will have to be
stopped separately.
Possibly a bug, I might want to inspect the TLS and JHS after shutting down
YARN, or I may just want to update the RM (Myriad is actively being
developed).
Might be worth reinvestigating at a later date. One pro I could see is by
setting the JHS and TLS in Myriad, it may be easier to assign them dynamic
ports via Mesos, but this would be a lot of added complexity for updating
the configs of running NMs if a JHS or TLS was reassigned. I don't think
this is something to address right now as it adds complexity which I think
can be avoided.
Darin
Hello here,
As we discussed last meeting I have wrote a small doc on pros/cons and
design suggestions for running JHS as MyriadTask.
https://docs.google.com/document/d/1TGGU2bn1cVPSmHeriNrtkAEhdvsRKcFtuWRaaGp9bSY/edit?usp=sharing
Also commented on: https://github.com/mesos/myriad/issues/60
Comments, suggestions, objections are welcome.
I am trying to wrap up design this week, so I could go one way or the other
starting next week, as I will be gone for 2 weeks, starting Aug 1st.
Thanks,Yuliya
