Re: [DISCUSS] official docker image(s) for hadoop

2017-09-22 Thread Marton, Elek 

Thanks all the feedbacks.

I created an issue:
https://issues.apache.org/jira/browse/HADOOP-14898

Let's continue the discussion there.

Thanks,
Marton

On 09/08/2017 02:45 PM, Marton, Elek wrote:


TL;DR: I propose to create official hadoop images and upload them to the 
dockerhub.


GOAL/SCOPE: I would like improve the existing documentation with 
easy-to-use docker based recipes to start hadoop clusters with various 
configuration.


The images also could be used to test experimental features. For example 
ozone could be tested easily with these compose file and configuration:


https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6

Or even the configuration could be included in the compose file:

https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml 



I would like to create separated example compose files for federation, 
ha, metrics usage, etc. to make it easier to try out and understand the 
features.


CONTEXT: There is an existing Jira 
https://issues.apache.org/jira/browse/HADOOP-13397
But it’s about a tool to generate production quality docker images 
(multiple types, in a flexible way). If no objections, I will create a 
separated issue to create simplified docker images for rapid prototyping 
and investigating new features. And register the branch to the dockerhub 
to create the images automatically.


MY BACKGROUND: I am working with docker based hadoop/spark clusters 
quite a while and run them succesfully in different environments 
(kubernetes, docker-swarm, nomad-based scheduling, etc.) My work is 
available from here: https://github.com/flokkr but they could handle 
more complex use cases (eg. instrumenting java processes with btrace, or 
read/reload configuration from consul).
  And IMHO in the official hadoop documentation it’s better to suggest 
to use official apache docker images and not external ones (which could 
be changed).


Please let me know if you have any comments.

Marton

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Mingliang Liu 
> It would be very helpful for testing the RC.
For testing and voting, I have been using docker containers for a while, see 
code at: https://github.com/weiqingy/caochong 


> TL;DR: I propose to create official hadoop images and upload them to the 
> dockerhub
I’m +1 on this idea. The “official” docker image basically means a commitment 
to maintain well documented and broadly tested images, which seems not a burden 
to us.

Ceph has a community docker project https://github.com/ceph/ceph-docker 
 and I think our scope here is similar to 
it.

Mingliang

> On Sep 13, 2017, at 11:39 AM, Yufei Gu  wrote:
> 
> It would be very helpful for testing the RC. To vote a RC, committers and
> PMCs usually spend lots of time to compile, deploy the RC, do several
> sanity tests, then +1 for the RC. The docker image potentially saves the
> compilation and deployment time, and people can do more tests.
> 
> Best,
> 
> Yufei
> 
> On Wed, Sep 13, 2017 at 11:19 AM, Wangda Tan  wrote:
> 
>> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
>> super helpful!
>> 
>> Thanks,
>> Wangda
>> 
>> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
>> miklos.szeg...@cloudera.com> wrote:
>> 
>>> Marton, thank you for working on this. I think Official Docker images for
>>> Hadoop would be very useful for a lot of reasons. I think that it is
>> better
>>> to have a coordinated effort with production ready base images with
>>> dependent images for prototyping. Does anyone else have an opinion about
>>> this?
>>> 
>>> Thank you,
>>> Miklos
>>> 
>>> On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
>>> 
 
 TL;DR: I propose to create official hadoop images and upload them to
>> the
 dockerhub.
 
 GOAL/SCOPE: I would like improve the existing documentation with
 easy-to-use docker based recipes to start hadoop clusters with various
 configuration.
 
 The images also could be used to test experimental features. For
>> example
 ozone could be tested easily with these compose file and configuration:
 
 https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
 
 Or even the configuration could be included in the compose file:
 
 https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
 ker-compose.yaml
 
 I would like to create separated example compose files for federation,
>>> ha,
 metrics usage, etc. to make it easier to try out and understand the
 features.
 
 CONTEXT: There is an existing Jira https://issues.apache.org/jira
 /browse/HADOOP-13397
 But it’s about a tool to generate production quality docker images
 (multiple types, in a flexible way). If no objections, I will create a
 separated issue to create simplified docker images for rapid
>> prototyping
 and investigating new features. And register the branch to the
>> dockerhub
>>> to
 create the images automatically.
 
 MY BACKGROUND: I am working with docker based hadoop/spark clusters
>> quite
 a while and run them succesfully in different environments (kubernetes,
 docker-swarm, nomad-based scheduling, etc.) My work is available from
>>> here:
 https://github.com/flokkr but they could handle more complex use cases
 (eg. instrumenting java processes with btrace, or read/reload
>>> configuration
 from consul).
 And IMHO in the official hadoop documentation it’s better to suggest
>> to
 use official apache docker images and not external ones (which could be
 changed).
 
 Please let me know if you have any comments.
 
 Marton
 
 -
 To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
 For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
 
 
>>> 
>>

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Yufei Gu 
It would be very helpful for testing the RC. To vote a RC, committers and
PMCs usually spend lots of time to compile, deploy the RC, do several
sanity tests, then +1 for the RC. The docker image potentially saves the
compilation and deployment time, and people can do more tests.

Best,

Yufei

On Wed, Sep 13, 2017 at 11:19 AM, Wangda Tan  wrote:

> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
> super helpful!
>
> Thanks,
> Wangda
>
> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
> miklos.szeg...@cloudera.com> wrote:
>
> > Marton, thank you for working on this. I think Official Docker images for
> > Hadoop would be very useful for a lot of reasons. I think that it is
> better
> > to have a coordinated effort with production ready base images with
> > dependent images for prototyping. Does anyone else have an opinion about
> > this?
> >
> > Thank you,
> > Miklos
> >
> > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
> >
> > >
> > > TL;DR: I propose to create official hadoop images and upload them to
> the
> > > dockerhub.
> > >
> > > GOAL/SCOPE: I would like improve the existing documentation with
> > > easy-to-use docker based recipes to start hadoop clusters with various
> > > configuration.
> > >
> > > The images also could be used to test experimental features. For
> example
> > > ozone could be tested easily with these compose file and configuration:
> > >
> > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> > >
> > > Or even the configuration could be included in the compose file:
> > >
> > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > > ker-compose.yaml
> > >
> > > I would like to create separated example compose files for federation,
> > ha,
> > > metrics usage, etc. to make it easier to try out and understand the
> > > features.
> > >
> > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > > /browse/HADOOP-13397
> > > But it’s about a tool to generate production quality docker images
> > > (multiple types, in a flexible way). If no objections, I will create a
> > > separated issue to create simplified docker images for rapid
> prototyping
> > > and investigating new features. And register the branch to the
> dockerhub
> > to
> > > create the images automatically.
> > >
> > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
> quite
> > > a while and run them succesfully in different environments (kubernetes,
> > > docker-swarm, nomad-based scheduling, etc.) My work is available from
> > here:
> > > https://github.com/flokkr but they could handle more complex use cases
> > > (eg. instrumenting java processes with btrace, or read/reload
> > configuration
> > > from consul).
> > >  And IMHO in the official hadoop documentation it’s better to suggest
> to
> > > use official apache docker images and not external ones (which could be
> > > changed).
> > >
> > > Please let me know if you have any comments.
> > >
> > > Marton
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Bharat Viswanadham 
+1 (non-binding)
It would be really nice to have Docker to try different features of Hadoop 
(like HA, Federation Enabled, Erasure coding…), which will helpful for both 
developers and users.


Thanks,
Bharat


On 9/13/17, 11:31 AM, "Eric Badger"  wrote:

+1 definitely think an official Hadoop docker image (possibly 1 per major
or minor release) would be a positive both for contributors and for users
of Hadoop.

Eric

On Wed, Sep 13, 2017 at 1:19 PM, Wangda Tan  wrote:

> +1 to add Hadoop docker image for easier testing / prototyping, it gonna 
be
> super helpful!
>
> Thanks,
> Wangda
>
> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
> miklos.szeg...@cloudera.com> wrote:
>
> > Marton, thank you for working on this. I think Official Docker images 
for
> > Hadoop would be very useful for a lot of reasons. I think that it is
> better
> > to have a coordinated effort with production ready base images with
> > dependent images for prototyping. Does anyone else have an opinion about
> > this?
> >
> > Thank you,
> > Miklos
> >
> > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
> >
> > >
> > > TL;DR: I propose to create official hadoop images and upload them to
> the
> > > dockerhub.
> > >
> > > GOAL/SCOPE: I would like improve the existing documentation with
> > > easy-to-use docker based recipes to start hadoop clusters with various
> > > configuration.
> > >
> > > The images also could be used to test experimental features. For
> example
> > > ozone could be tested easily with these compose file and 
configuration:
> > >
> > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> > >
> > > Or even the configuration could be included in the compose file:
> > >
> > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > > ker-compose.yaml
> > >
> > > I would like to create separated example compose files for federation,
> > ha,
> > > metrics usage, etc. to make it easier to try out and understand the
> > > features.
> > >
> > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > > /browse/HADOOP-13397
> > > But it’s about a tool to generate production quality docker images
> > > (multiple types, in a flexible way). If no objections, I will create a
> > > separated issue to create simplified docker images for rapid
> prototyping
> > > and investigating new features. And register the branch to the
> dockerhub
> > to
> > > create the images automatically.
> > >
> > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
> quite
> > > a while and run them succesfully in different environments 
(kubernetes,
> > > docker-swarm, nomad-based scheduling, etc.) My work is available from
> > here:
> > > https://github.com/flokkr but they could handle more complex use cases
> > > (eg. instrumenting java processes with btrace, or read/reload
> > configuration
> > > from consul).
> > >  And IMHO in the official hadoop documentation it’s better to suggest
> to
> > > use official apache docker images and not external ones (which could 
be
> > > changed).
> > >
> > > Please let me know if you have any comments.
> > >
> > > Marton
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Eric Badger 
+1 definitely think an official Hadoop docker image (possibly 1 per major
or minor release) would be a positive both for contributors and for users
of Hadoop.

Eric

On Wed, Sep 13, 2017 at 1:19 PM, Wangda Tan  wrote:

> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
> super helpful!
>
> Thanks,
> Wangda
>
> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
> miklos.szeg...@cloudera.com> wrote:
>
> > Marton, thank you for working on this. I think Official Docker images for
> > Hadoop would be very useful for a lot of reasons. I think that it is
> better
> > to have a coordinated effort with production ready base images with
> > dependent images for prototyping. Does anyone else have an opinion about
> > this?
> >
> > Thank you,
> > Miklos
> >
> > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
> >
> > >
> > > TL;DR: I propose to create official hadoop images and upload them to
> the
> > > dockerhub.
> > >
> > > GOAL/SCOPE: I would like improve the existing documentation with
> > > easy-to-use docker based recipes to start hadoop clusters with various
> > > configuration.
> > >
> > > The images also could be used to test experimental features. For
> example
> > > ozone could be tested easily with these compose file and configuration:
> > >
> > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> > >
> > > Or even the configuration could be included in the compose file:
> > >
> > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > > ker-compose.yaml
> > >
> > > I would like to create separated example compose files for federation,
> > ha,
> > > metrics usage, etc. to make it easier to try out and understand the
> > > features.
> > >
> > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > > /browse/HADOOP-13397
> > > But it’s about a tool to generate production quality docker images
> > > (multiple types, in a flexible way). If no objections, I will create a
> > > separated issue to create simplified docker images for rapid
> prototyping
> > > and investigating new features. And register the branch to the
> dockerhub
> > to
> > > create the images automatically.
> > >
> > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
> quite
> > > a while and run them succesfully in different environments (kubernetes,
> > > docker-swarm, nomad-based scheduling, etc.) My work is available from
> > here:
> > > https://github.com/flokkr but they could handle more complex use cases
> > > (eg. instrumenting java processes with btrace, or read/reload
> > configuration
> > > from consul).
> > >  And IMHO in the official hadoop documentation it’s better to suggest
> to
> > > use official apache docker images and not external ones (which could be
> > > changed).
> > >
> > > Please let me know if you have any comments.
> > >
> > > Marton
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Wangda Tan 
+1 to add Hadoop docker image for easier testing / prototyping, it gonna be
super helpful!

Thanks,
Wangda

On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
miklos.szeg...@cloudera.com> wrote:

> Marton, thank you for working on this. I think Official Docker images for
> Hadoop would be very useful for a lot of reasons. I think that it is better
> to have a coordinated effort with production ready base images with
> dependent images for prototyping. Does anyone else have an opinion about
> this?
>
> Thank you,
> Miklos
>
> On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
>
> >
> > TL;DR: I propose to create official hadoop images and upload them to the
> > dockerhub.
> >
> > GOAL/SCOPE: I would like improve the existing documentation with
> > easy-to-use docker based recipes to start hadoop clusters with various
> > configuration.
> >
> > The images also could be used to test experimental features. For example
> > ozone could be tested easily with these compose file and configuration:
> >
> > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> >
> > Or even the configuration could be included in the compose file:
> >
> > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > ker-compose.yaml
> >
> > I would like to create separated example compose files for federation,
> ha,
> > metrics usage, etc. to make it easier to try out and understand the
> > features.
> >
> > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > /browse/HADOOP-13397
> > But it’s about a tool to generate production quality docker images
> > (multiple types, in a flexible way). If no objections, I will create a
> > separated issue to create simplified docker images for rapid prototyping
> > and investigating new features. And register the branch to the dockerhub
> to
> > create the images automatically.
> >
> > MY BACKGROUND: I am working with docker based hadoop/spark clusters quite
> > a while and run them succesfully in different environments (kubernetes,
> > docker-swarm, nomad-based scheduling, etc.) My work is available from
> here:
> > https://github.com/flokkr but they could handle more complex use cases
> > (eg. instrumenting java processes with btrace, or read/reload
> configuration
> > from consul).
> >  And IMHO in the official hadoop documentation it’s better to suggest to
> > use official apache docker images and not external ones (which could be
> > changed).
> >
> > Please let me know if you have any comments.
> >
> > Marton
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>