Re: Multiple node Hadoop cluster on Docker for test and debugging

[Sasaki Kai]

Roman, Zhe

Thanks for the response!

I was not familiar with Apache BigTop. I also try to use it. Thanks!

Kai Sasaki

> On 2016 Nov 10, at 9:07, Roman Shaposhnik  wrote:
> 
> On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang  
> wrote:
>> Nice work Kai! Quick comment: have you considered Bigtop? It allows
>> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
>> a custom built Hadoop (from the instructions I read, it builds from a
>> certain git branch; I'm not sure if that customizable).
> 
> That's perfectly customizable. What's cool about Bigtop is that it allows
> you to provision not just Hadoop but tons of other big data projects since
> it is all driven by the set of Puppet recipes. The very same set of Puppet
> recipes that also drive Canonical's Juju:
>https://jujucharms.com/big-data
> 
> Thanks,
> Roman.
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

[Roman Shaposhnik]

On Wed, Nov 9, 2016 at 9:30 AM, Zhe Zhang  wrote:
> Nice work Kai! Quick comment: have you considered Bigtop? It allows
> creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
> a custom built Hadoop (from the instructions I read, it builds from a
> certain git branch; I'm not sure if that customizable).

That's perfectly customizable. What's cool about Bigtop is that it allows
you to provision not just Hadoop but tons of other big data projects since
it is all driven by the set of Puppet recipes. The very same set of Puppet
recipes that also drive Canonical's Juju:
https://jujucharms.com/big-data

Thanks,
Roman.

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Multiple node Hadoop cluster on Docker for test and debugging

[Mingliang Liu]

I know a similar tool; it has Ambari and Spark integration as well.

https://github.com/weiqingy/caochong 

Thanks,

L

> On Nov 9, 2016, at 5:38 AM, Sasaki Kai  wrote:
> 
> Hi Hadoop developers
> 
> The other day I created a tool for launching multiple node hadoop cluster on 
> docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source 
> code. 
> It is useful for testing and debugging. Actually I often use it before 
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster 
> 
> 
> And I also updated to build the latest trunk image automatically and upload 
> onto Docker Hub.
> So you can easily check and test the latest trunk branch in the environment 
> which is more close to actual usage.
> 
> If you already installed docker and docker-compose, what needed is 
> docker-compose.yml like this.
> 
> version: '2'
> 
> services:
>  master:
>image: lewuathe/hadoop-master
>ports:
>  - "9870:9870"
>  - "8088:8088"
>  - "19888:19888"
>  - "8188:8188"
>container_name: "master"
>  slave1:
>image: lewuathe/hadoop-slave
>container_name: "slave1"
>depends_on:
>  - master
>ports:
>  - "9901:9864"
>  - "8041:8042"
>  slave2:
>image: lewuathe/hadoop-slave
>container_name: "slave2"
>depends_on:
>  - master
>ports:
>  - "9902:9864"
>  - "8042:8042"
> 
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md 
> 
> 
> I would be glad if you use this tool for developing and debugging and make 
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
> 
> 
> Kai Sasaki
> mail: lewua...@me.com 
> github: https://github.com/Lewuathe 
> 
> 

Re: Multiple node Hadoop cluster on Docker for test and debugging

[Zhe Zhang]

Nice work Kai! Quick comment: have you considered Bigtop? It allows
creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
a custom built Hadoop (from the instructions I read, it builds from a
certain git branch; I'm not sure if that customizable).

On Wed, Nov 9, 2016 at 5:39 AM Sasaki Kai  wrote:

> Hi Hadoop developers
>
> The other day I created a tool for launching multiple node hadoop cluster
> on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source
> code.
> It is useful for testing and debugging. Actually I often use it before
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <
> https://github.com/Lewuathe/docker-hadoop-cluster>
>
> And I also updated to build the latest trunk image automatically and
> upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the
> environment which is more close to actual usage.
>
> If you already installed docker and docker-compose, what needed is
> docker-compose.yml like this.
>
> version: '2'
>
> services:
>   master:
> image: lewuathe/hadoop-master
> ports:
>   - "9870:9870"
>   - "8088:8088"
>   - "19888:19888"
>   - "8188:8188"
> container_name: "master"
>   slave1:
> image: lewuathe/hadoop-slave
> container_name: "slave1"
> depends_on:
>   - master
> ports:
>   - "9901:9864"
>   - "8041:8042"
>   slave2:
> image: lewuathe/hadoop-slave
> container_name: "slave2"
> depends_on:
>   - master
> ports:
>   - "9902:9864"
>   - "8042:8042"
>
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
>
> I would be glad if you use this tool for developing and debugging and make
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
>
> 
> Kai Sasaki
> mail: lewua...@me.com 
> github: https://github.com/Lewuathe 
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Multiple node Hadoop cluster on Docker for test and debugging

[Sasaki Kai]

Hi Hadoop developers

The other day I created a tool for launching multiple node hadoop cluster on 
docker container.
You can easily launch multiple node hadoop cluster from your Hadoop source 
code. 
It is useful for testing and debugging. Actually I often use it before 
submitting a patch to Hadoop project.
https://github.com/Lewuathe/docker-hadoop-cluster 


And I also updated to build the latest trunk image automatically and upload 
onto Docker Hub.
So you can easily check and test the latest trunk branch in the environment 
which is more close to actual usage.

If you already installed docker and docker-compose, what needed is 
docker-compose.yml like this.

version: '2'

services:
  master:
image: lewuathe/hadoop-master
ports:
  - "9870:9870"
  - "8088:8088"
  - "19888:19888"
  - "8188:8188"
container_name: "master"
  slave1:
image: lewuathe/hadoop-slave
container_name: "slave1"
depends_on:
  - master
ports:
  - "9901:9864"
  - "8041:8042"
  slave2:
image: lewuathe/hadoop-slave
container_name: "slave2"
depends_on:
  - master
ports:
  - "9902:9864"
  - "8042:8042"

The usage in detail is described in the repository.
https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md 


I would be glad if you use this tool for developing and debugging and make our 
development more efficient.
Please give me any feedbacks to me. Thanks you!


Kai Sasaki
mail: lewua...@me.com 
github: https://github.com/Lewuathe