答复: 答复: Hello Guys

[陈泳宇]

Darin, 

 Thank you for your reply. As you mentioned that, Myriad NM is launched 
by Myriad Framework, while CM could be launching the NM in a totally different 
way. I think I should figure out the procedure first.

 Since Myriad could utilize resource from Mesos, and at the same time 
it is a YARN FairScheducler.  Although it's impractical to perfect integrate CM 
with Mesos，there still could be a chance to intergrate them under this specific 
scene.  With the help of Myriad, may be it is possible to share the resource 
between these two framework. 

 

 

Regards, 

Yongyu

发件人: Darin Johnson [mailto:dbjohnson1...@gmail.com] 
发送时间: 2016年4月27日 22:47
收件人: 陈泳宇; Dev
抄送: c...@linkernetworks.com; s...@linkernetworks.com; Sam Chen
主题: Re: 答复: Hello Guys

 

Yongyu,

 

Sounds like you've made some progress! I'm not sure you'll be able to get the 
cluster completely under CM management and the node managers launched by Myriad 
aren't running the Cloudera agent (I don't think we'll ever officially support 
that, but you could tweak the source to run it in the background, if you could 
make it a general process it might be a feature we'd add).  Though if you're 
launching the Resource Manager via CM I'd expect it to show up.  I haven't 
attempted to put my resource manager under the CM though.

 

Darin

 

On Tue, Apr 26, 2016 at 4:45 AM, 陈泳宇 <yc...@linkernetworks.com 
<mailto:yc...@linkernetworks.com> > wrote:

Hi Darin, This is Yongyu from Linkernetworks, I have proved that myriad can 
work well with CDH. Currently, I am also trying to myriad our hadoop cluster 
under the management of CM.

I noticed that you have already complete the mission, but I still got some 
problem here: The status shown on CM dashboard will not change…



Here are my steps:

1.   Add YARN service on CM dashboard..

2.   Stop the YARN service

3.   Cd to /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45

4.   Copy native libs 、myriad jars as well we the config files.

5.   Tar this package and upload it to hdfs.

6.   Start the resourcemanager.

Myriad service will be launched successfully, but nothing changes on the 
dashboard, which means the cluster is not under the management of CM..

 

 

Regards, 

Yongyu


Begin forwarded message:

From: Darin Johnson <dbjohnson1...@gmail.com <mailto:dbjohnson1...@gmail.com> >
Date: April 21, 2016 at 8:50:22 PM GMT+8
To: Dev <dev@myriad.incubator.apache.org 
<mailto:dev@myriad.incubator.apache.org> >
Subject: Re: Hello Guys
Reply-To: dev@myriad.incubator.apache.org 
<mailto:dev@myriad.incubator.apache.org> 

Hey Sam,

I'm already myriad alongside a CM managed hadoop cluster.  It's a little
hacky right now, I'm working on stream lining this, it may involve CM
and/or some docker integration.  Here are my current steps:

0. I strongly recommend pulling of master and building from source - it has
some really useful patches.  We're working on another release now (couple
weeks out though).

1. Let Cloudera Manager configure hdfs - it does a good job of this.

2. Grab a cloudera tar ball from here:
http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_eom.html#topic_3
(I've also just used apache-hadoop tarballs).

3. Extract the tar ball, copy the native libraries cdh install on your
system in hadoop-*/lib/native

4 cp myriad/myriad-*/build/libs/* hadoop-*/share/hadoop/yarn/lib

5 copy your hadoop configs to hadoop-*/etc/hadoop/

6 create a myriad-default-config.yml in hadoop-*/etc/hadoop follow
instructions on wiki for remote distribution for edits to
myriad-default-config.yml.  NB: don't enable yarn cgroups yet - I'm fixing
a bug.

7 chown -R root:root hadoop-*y

8 chown root:yarn hadoop-*/bin/container-executor ; chmod g+s
hadoop*/bin/container-executor

9 mv hadoop-<..> hadoop-myriad

10 tar -zxvf hadoop-myriad.tgz hadoop-myriad ; hadoop fs -put
hadoop-myriad.tgz /dist/

11 cd hadoop-myriad/ && sudo -u yarn bin/yarn resource manager

12 hit the web-ui at host:8192 and flexup some node managers

This has been pretty stable.

Alternatively, if you're running mesos and docker you could look at PR
https://github.com/apache/incubator-myriad/pull/64, it's still WIP but
avoids a lot of setup - would be happy to work with you as I document and
harden this feature for the 0.2.0 release.  Currently that runs off generic
hadoop put we could certainly create distribution specific dockerfiles.

Let me know if you need help.  Keep in mind this is still an alpha project,
so expect some issues. Would love to get feedback, use cases and feature
ideas.

Also a mesos tip: if your running services like hdfs outside of mesos you
should adjust you're mesos resources appropriately or you'll end up over
subscribed and processes will slow or die.

Thanks for trying myriad!

Darin


On Apr 21, 2016 4:22 AM, "rchen" <rc...@linkernetworks.com 
<mailto:rc...@lin

Re: 答复: Hello Guys

[Darin Johnson]

Yongyu,

Sounds like you've made some progress! I'm not sure you'll be able to get
the cluster completely under CM management and the node managers launched
by Myriad aren't running the Cloudera agent (I don't think we'll ever
officially support that, but you could tweak the source to run it in the
background, if you could make it a general process it might be a feature
we'd add).  Though if you're launching the Resource Manager via CM I'd
expect it to show up.  I haven't attempted to put my resource manager under
the CM though.

Darin

On Tue, Apr 26, 2016 at 4:45 AM, 陈泳宇 <yc...@linkernetworks.com> wrote:

> Hi Darin, This is Yongyu from Linkernetworks, I have proved that myriad
> can work well with CDH. Currently, I am also trying to myriad our hadoop
> cluster under the management of CM.
>
> I noticed that you have already complete the mission, but I still got some
> problem here: The status shown on CM dashboard will not change…
>
> Here are my steps:
>
> 1.   Add YARN service on CM dashboard..
>
> 2.   Stop the YARN service
>
> 3.   Cd to /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45
>
> 4.   Copy native libs 、myriad jars as well we the config files.
>
> 5.   Tar this package and upload it to hdfs.
>
> 6.   Start the resourcemanager.
>
> Myriad service will be launched successfully, but nothing changes on the
> dashboard, which means the cluster is not under the management of CM..
>
>
>
>
>
> Regards,
>
> Yongyu
>
>
> Begin forwarded message:
>
> *From:* Darin Johnson <dbjohnson1...@gmail.com>
> *Date:* April 21, 2016 at 8:50:22 PM GMT+8
> *To:* Dev <dev@myriad.incubator.apache.org>
> *Subject:* *Re: Hello Guys*
> *Reply-To:* dev@myriad.incubator.apache.org
>
> Hey Sam,
>
> I'm already myriad alongside a CM managed hadoop cluster.  It's a little
> hacky right now, I'm working on stream lining this, it may involve CM
> and/or some docker integration.  Here are my current steps:
>
> 0. I strongly recommend pulling of master and building from source - it has
> some really useful patches.  We're working on another release now (couple
> weeks out though).
>
> 1. Let Cloudera Manager configure hdfs - it does a good job of this.
>
> 2. Grab a cloudera tar ball from here:
>
> http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_eom.html#topic_3
> (I've also just used apache-hadoop tarballs).
>
> 3. Extract the tar ball, copy the native libraries cdh install on your
> system in hadoop-*/lib/native
>
> 4 cp myriad/myriad-*/build/libs/* hadoop-*/share/hadoop/yarn/lib
>
> 5 copy your hadoop configs to hadoop-*/etc/hadoop/
>
> 6 create a myriad-default-config.yml in hadoop-*/etc/hadoop follow
> instructions on wiki for remote distribution for edits to
> myriad-default-config.yml.  NB: don't enable yarn cgroups yet - I'm fixing
> a bug.
>
> 7 chown -R root:root hadoop-*y
>
> 8 chown root:yarn hadoop-*/bin/container-executor ; chmod g+s
> hadoop*/bin/container-executor
>
> 9 mv hadoop-<..> hadoop-myriad
>
> 10 tar -zxvf hadoop-myriad.tgz hadoop-myriad ; hadoop fs -put
> hadoop-myriad.tgz /dist/
>
> 11 cd hadoop-myriad/ && sudo -u yarn bin/yarn resource manager
>
> 12 hit the web-ui at host:8192 and flexup some node managers
>
> This has been pretty stable.
>
> Alternatively, if you're running mesos and docker you could look at PR
> https://github.com/apache/incubator-myriad/pull/64, it's still WIP but
> avoids a lot of setup - would be happy to work with you as I document and
> harden this feature for the 0.2.0 release.  Currently that runs off generic
> hadoop put we could certainly create distribution specific dockerfiles.
>
> Let me know if you need help.  Keep in mind this is still an alpha project,
> so expect some issues. Would love to get feedback, use cases and feature
> ideas.
>
> Also a mesos tip: if your running services like hdfs outside of mesos you
> should adjust you're mesos resources appropriately or you'll end up over
> subscribed and processes will slow or die.
>
> Thanks for trying myriad!
>
> Darin
>
>
> On Apr 21, 2016 4:22 AM, "rchen" <rc...@linkernetworks.com> wrote:
>
> Hi Guys,
> Currrently, we are working on Myriad with Cloudera distribution. And we
> have proved that Mesos and work with Yarn.
> However talking about CM, how to integrate ? Any comments is welcome.
> appreciated.
>
>
> Regards
> Sam
>
>

Re: Hello Guys

[Darin Johnson]

Hey Sam,

I'm already myriad alongside a CM managed hadoop cluster.  It's a little
hacky right now, I'm working on stream lining this, it may involve CM
and/or some docker integration.  Here are my current steps:

0. I strongly recommend pulling of master and building from source - it has
some really useful patches.  We're working on another release now (couple
weeks out though).

1. Let Cloudera Manager configure hdfs - it does a good job of this.

2. Grab a cloudera tar ball from here:
http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_eom.html#topic_3
(I've also just used apache-hadoop tarballs).

3. Extract the tar ball, copy the native libraries cdh install on your
system in hadoop-*/lib/native

4 cp myriad/myriad-*/build/libs/* hadoop-*/share/hadoop/yarn/lib

5 copy your hadoop configs to hadoop-*/etc/hadoop/

6 create a myriad-default-config.yml in hadoop-*/etc/hadoop follow
instructions on wiki for remote distribution for edits to
myriad-default-config.yml.  NB: don't enable yarn cgroups yet - I'm fixing
a bug.

7 chown -R root:root hadoop-*y

8 chown root:yarn hadoop-*/bin/container-executor ; chmod g+s
hadoop*/bin/container-executor

9 mv hadoop-<..> hadoop-myriad

10 tar -zxvf hadoop-myriad.tgz hadoop-myriad ; hadoop fs -put
hadoop-myriad.tgz /dist/

11 cd hadoop-myriad/ && sudo -u yarn bin/yarn resource manager

12 hit the web-ui at host:8192 and flexup some node managers

This has been pretty stable.

Alternatively, if you're running mesos and docker you could look at PR
https://github.com/apache/incubator-myriad/pull/64, it's still WIP but
avoids a lot of setup - would be happy to work with you as I document and
harden this feature for the 0.2.0 release.  Currently that runs off generic
hadoop put we could certainly create distribution specific dockerfiles.

Let me know if you need help.  Keep in mind this is still an alpha project,
so expect some issues. Would love to get feedback, use cases and feature
ideas.

Also a mesos tip: if your running services like hdfs outside of mesos you
should adjust you're mesos resources appropriately or you'll end up over
subscribed and processes will slow or die.

Thanks for trying myriad!

Darin


On Apr 21, 2016 4:22 AM, "rchen"  wrote:

Hi Guys,
Currrently, we are working on Myriad with Cloudera distribution. And we
have proved that Mesos and work with Yarn.
However talking about CM, how to integrate ? Any comments is welcome.
appreciated.


Regards
Sam