Darin,
Thank you for your reply. As you mentioned that, Myriad NM is launched
by Myriad Framework, while CM could be launching the NM in a totally different
way. I think I should figure out the procedure first.
Since Myriad could utilize resource from Mesos, and at the same time
it is a YARN FairScheducler. Although it's impractical to perfect integrate CM
with Mesos,there still could be a chance to intergrate them under this specific
scene. With the help of Myriad, may be it is possible to share the resource
between these two framework.
Regards,
Yongyu
发件人: Darin Johnson [mailto:dbjohnson1...@gmail.com]
发送时间: 2016年4月27日 22:47
收件人: 陈泳宇; Dev
抄送: c...@linkernetworks.com; s...@linkernetworks.com; Sam Chen
主题: Re: 答复: Hello Guys
Yongyu,
Sounds like you've made some progress! I'm not sure you'll be able to get the
cluster completely under CM management and the node managers launched by Myriad
aren't running the Cloudera agent (I don't think we'll ever officially support
that, but you could tweak the source to run it in the background, if you could
make it a general process it might be a feature we'd add). Though if you're
launching the Resource Manager via CM I'd expect it to show up. I haven't
attempted to put my resource manager under the CM though.
Darin
On Tue, Apr 26, 2016 at 4:45 AM, 陈泳宇 <yc...@linkernetworks.com
<mailto:yc...@linkernetworks.com> > wrote:
Hi Darin, This is Yongyu from Linkernetworks, I have proved that myriad can
work well with CDH. Currently, I am also trying to myriad our hadoop cluster
under the management of CM.
I noticed that you have already complete the mission, but I still got some
problem here: The status shown on CM dashboard will not change…
Here are my steps:
1. Add YARN service on CM dashboard..
2. Stop the YARN service
3. Cd to /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45
4. Copy native libs 、myriad jars as well we the config files.
5. Tar this package and upload it to hdfs.
6. Start the resourcemanager.
Myriad service will be launched successfully, but nothing changes on the
dashboard, which means the cluster is not under the management of CM..
Regards,
Yongyu
Begin forwarded message:
From: Darin Johnson <dbjohnson1...@gmail.com <mailto:dbjohnson1...@gmail.com> >
Date: April 21, 2016 at 8:50:22 PM GMT+8
To: Dev <dev@myriad.incubator.apache.org
<mailto:dev@myriad.incubator.apache.org> >
Subject: Re: Hello Guys
Reply-To: dev@myriad.incubator.apache.org
<mailto:dev@myriad.incubator.apache.org>
Hey Sam,
I'm already myriad alongside a CM managed hadoop cluster. It's a little
hacky right now, I'm working on stream lining this, it may involve CM
and/or some docker integration. Here are my current steps:
0. I strongly recommend pulling of master and building from source - it has
some really useful patches. We're working on another release now (couple
weeks out though).
1. Let Cloudera Manager configure hdfs - it does a good job of this.
2. Grab a cloudera tar ball from here:
http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_eom.html#topic_3
(I've also just used apache-hadoop tarballs).
3. Extract the tar ball, copy the native libraries cdh install on your
system in hadoop-*/lib/native
4 cp myriad/myriad-*/build/libs/* hadoop-*/share/hadoop/yarn/lib
5 copy your hadoop configs to hadoop-*/etc/hadoop/
6 create a myriad-default-config.yml in hadoop-*/etc/hadoop follow
instructions on wiki for remote distribution for edits to
myriad-default-config.yml. NB: don't enable yarn cgroups yet - I'm fixing
a bug.
7 chown -R root:root hadoop-*y
8 chown root:yarn hadoop-*/bin/container-executor ; chmod g+s
hadoop*/bin/container-executor
9 mv hadoop-<..> hadoop-myriad
10 tar -zxvf hadoop-myriad.tgz hadoop-myriad ; hadoop fs -put
hadoop-myriad.tgz /dist/
11 cd hadoop-myriad/ && sudo -u yarn bin/yarn resource manager
12 hit the web-ui at host:8192 and flexup some node managers
This has been pretty stable.
Alternatively, if you're running mesos and docker you could look at PR
https://github.com/apache/incubator-myriad/pull/64, it's still WIP but
avoids a lot of setup - would be happy to work with you as I document and
harden this feature for the 0.2.0 release. Currently that runs off generic
hadoop put we could certainly create distribution specific dockerfiles.
Let me know if you need help. Keep in mind this is still an alpha project,
so expect some issues. Would love to get feedback, use cases and feature
ideas.
Also a mesos tip: if your running services like hdfs outside of mesos you
should adjust you're mesos resources appropriately or you'll end up over
subscribed and processes will slow or die.
Thanks for trying myriad!
Darin
On Apr 21, 2016 4:22 AM, "rchen" <rc...@linkernetworks.com
<mailto:rc...@lin