Re: Create multiarch docker images for bigtop-puppet/slaves

[Jun HE]

Thanks for pointing the resources and suggestions , Roman, :)

Regards the multi-arch, my intension is to simplify the images name, so the
actual underlying things (arch-depent images) are transparent to Bigtop
users/developers. Say, they can just issue same cmd: docker pull
bigtop/puppet:1.5-centos-7, on any platform, and then start doing coll
stuffs.

And another puprpose is to simplify provision/test process. As you can see
there are some arch related yaml files in
https://github.com/apache/bigtop/tree/master/provisioner/docker.
If multi-arch is enabled, we can remove those config-*-aarch64.yaml files,
and use just one unified yaml file for certain distro/packages. Also I
think this could help devOps guys if they're using Bigtop to manage
deployment.


2018-05-29 14:04 GMT+08:00 Roman Shaposhnik :

> Here's a few points from the peanut gallery: in general I find
> Docker multiarch pretty useful, but at the same time its not like
> having a schema like:
> bigtop/:-- version>-
> wouldn't achieve the same. I'd say go with what feels simpler.
>
> As for scripts that create those -- that's just puppet in:
> https://github.com/apache/bigtop/tree/master/bigtop_toolchain
>
> So your Dockerfile is super-straightforward -- start with the base,
> add puppet via:
>https://github.com/apache/bigtop/blob/master/bigtop_
> toolchain/bin/puppetize.sh
> and run the puppet.
>
> In fact, I believe that this bakes the images (unless it bitrotted
> somehow):
>https://github.com/apache/bigtop/blob/master/docker/
> bigtop-slaves/build.sh
>
>
> Thanks,
> Roman.
>
> On Sun, May 27, 2018 at 7:55 PM, Jun HE  wrote:
> > Thanks for comments, Evans.
> >
> > There should not be much efforts to enable multi-arch. The question is
> how
> > we plan to support it in Bigtop.
> >
> > I can think of several items to discuss for this feature:
> > 1. Docker images name conventions:
> > For this, I'd suggest to using this:
> > * for actual underlying docker image running on specific platform,
> name
> > it as:
> > bigtop/:-- version>-,
> > say: bigtop/puppet:trunk-debian-9-amd64,
> > bigtop/slaves:1.3.0-fedora-26-aarch64
> > * for virtual image, name it as: bigtop/:
> > --,
> > say:  bigtop/puppet:trunk-debian-9,  bigtop/slaves:1.3.0-fedora-26
> > 2. How to create these images/manifests
> > I didn't find any cmds/scripts in Bigtop source to do the docker hub
> > uploading work. So I guess we're doing it manually for now. If we're
> going
> > to use multi-arch, we need to upload the virtual image's manifest file to
> > docker hub besides those arch dependent images.
> > Docker cli has a cmd to do such work: docker manifest. But it is not
> in
> > distros' docker releases. And it is marked as "experimental" feature,
> user
> > needs to change docker settings to use it. See
> > https://docs.docker.com/edge/engine/reference/commandline/
> manifest/#description.
> > I created a shell script version (
> > https://github.com/JunHe77/docker-manifest-sh) to do the same thing, use
> > same cmd syntax. Hope this can help.
> > 3. Update existing settings/yaml files to use the virutal distro image.
> > Once the images/manifests are ready in docker hub, we can update
> > existing settings/yaml files to use "virtual" bigtop images. This should
> > take too much efforts. And we'll get unified, beautiful code for
> different
> > archs.
> >
> > Pls raise your thoughts, questions, and etc. Once we settle these
> > workflows/specs down, I'm sure it will be ready before Sept.
> >
> > Thanks.
> >
> > 2018-05-28 1:45 GMT+08:00 Evans Ye :
> >
> >> I'd say this is a great idea.
> >> The only reason we haven't adopt multi-arch is because we started to
> adopt
> >> docker at early stage so there's no such feature then.
> >>
> >> Jun can you roughly estimate the effort? If we're going for it, we
> better
> >> have it ready in 1.3.0 before ApacheCon which is around September.
> >>
> >>
> >> 2018-05-17 17:19 GMT+08:00 Naresh Bhat :
> >>
> >> > On 17 May 2018 at 13:21, Jun HE  wrote:
> >> > > Hi, guys,
> >> > >
> >> > > As bigtop has simplified the way to create docker images for
> distros,
> >> by
> >> > > BIGTOP-2922, I'm thinking of multi-arch docker images support to
> ease
> >> > > building/test/development.
> >> > >
> >> > > Multi-arch feature has been supported since 2017, you may check this
> >> > link:
> >> > > https://blog.docker.com/2017/11/multi-arch-all-the-things/
> >> > > In simple words, multi-arch is a virtual image which contains
> several
> >> > > architecture dependent images. When user try to pull this virtual
> image
> >> > > dockhub will send back the actual image based on the architecture
> he is
> >> > > using.
> >> > >
> >> > > Use multi-arch will help:
> >> > > * development: user don't have to issue different cmd to pull
> >> > puppet/slaves
> >> > > images, "docker pull bigtop/slaves:trunk-debian-9" is enough
> instead of
> >> > "docker
> >> > > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull
> 

Re: Create multiarch docker images for bigtop-puppet/slaves

[Roman Shaposhnik]

Here's a few points from the peanut gallery: in general I find
Docker multiarch pretty useful, but at the same time its not like
having a schema like:
bigtop/:---
wouldn't achieve the same. I'd say go with what feels simpler.

As for scripts that create those -- that's just puppet in:
https://github.com/apache/bigtop/tree/master/bigtop_toolchain

So your Dockerfile is super-straightforward -- start with the base,
add puppet via:
   
https://github.com/apache/bigtop/blob/master/bigtop_toolchain/bin/puppetize.sh
and run the puppet.

In fact, I believe that this bakes the images (unless it bitrotted somehow):
   https://github.com/apache/bigtop/blob/master/docker/bigtop-slaves/build.sh


Thanks,
Roman.

On Sun, May 27, 2018 at 7:55 PM, Jun HE  wrote:
> Thanks for comments, Evans.
>
> There should not be much efforts to enable multi-arch. The question is how
> we plan to support it in Bigtop.
>
> I can think of several items to discuss for this feature:
> 1. Docker images name conventions:
> For this, I'd suggest to using this:
> * for actual underlying docker image running on specific platform, name
> it as:
> bigtop/:---,
> say: bigtop/puppet:trunk-debian-9-amd64,
> bigtop/slaves:1.3.0-fedora-26-aarch64
> * for virtual image, name it as: bigtop/:
> --,
> say:  bigtop/puppet:trunk-debian-9,  bigtop/slaves:1.3.0-fedora-26
> 2. How to create these images/manifests
> I didn't find any cmds/scripts in Bigtop source to do the docker hub
> uploading work. So I guess we're doing it manually for now. If we're going
> to use multi-arch, we need to upload the virtual image's manifest file to
> docker hub besides those arch dependent images.
> Docker cli has a cmd to do such work: docker manifest. But it is not in
> distros' docker releases. And it is marked as "experimental" feature, user
> needs to change docker settings to use it. See
> https://docs.docker.com/edge/engine/reference/commandline/manifest/#description.
> I created a shell script version (
> https://github.com/JunHe77/docker-manifest-sh) to do the same thing, use
> same cmd syntax. Hope this can help.
> 3. Update existing settings/yaml files to use the virutal distro image.
> Once the images/manifests are ready in docker hub, we can update
> existing settings/yaml files to use "virtual" bigtop images. This should
> take too much efforts. And we'll get unified, beautiful code for different
> archs.
>
> Pls raise your thoughts, questions, and etc. Once we settle these
> workflows/specs down, I'm sure it will be ready before Sept.
>
> Thanks.
>
> 2018-05-28 1:45 GMT+08:00 Evans Ye :
>
>> I'd say this is a great idea.
>> The only reason we haven't adopt multi-arch is because we started to adopt
>> docker at early stage so there's no such feature then.
>>
>> Jun can you roughly estimate the effort? If we're going for it, we better
>> have it ready in 1.3.0 before ApacheCon which is around September.
>>
>>
>> 2018-05-17 17:19 GMT+08:00 Naresh Bhat :
>>
>> > On 17 May 2018 at 13:21, Jun HE  wrote:
>> > > Hi, guys,
>> > >
>> > > As bigtop has simplified the way to create docker images for distros,
>> by
>> > > BIGTOP-2922, I'm thinking of multi-arch docker images support to ease
>> > > building/test/development.
>> > >
>> > > Multi-arch feature has been supported since 2017, you may check this
>> > link:
>> > > https://blog.docker.com/2017/11/multi-arch-all-the-things/
>> > > In simple words, multi-arch is a virtual image which contains several
>> > > architecture dependent images. When user try to pull this virtual image
>> > > dockhub will send back the actual image based on the architecture he is
>> > > using.
>> > >
>> > > Use multi-arch will help:
>> > > * development: user don't have to issue different cmd to pull
>> > puppet/slaves
>> > > images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of
>> > "docker
>> > > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull
>> > > bigtop/slaves:trunk-debian-9-aarch64"
>> > > * test: unified yaml file for same distro. refer example in
>> > > provisioner/docker/config_*
>> > >
>> > > So how do you think? :)
>> >
>> > Thanks Jun for initiating the thread.
>> > +1 this makes sense. Is there any reason why we have not consider for
>> > using multi-arch images ? If anybody explains advantage/disadvantage
>> > of using multi-arch docker image in bigtop, it will help us to
>> > understand in a better way.
>> >
>> > >
>> > > Regards,
>> > >
>> > > Jun
>> >
>>

Re: Create multiarch docker images for bigtop-puppet/slaves

[Jun HE]

Thanks for comments, Evans.

There should not be much efforts to enable multi-arch. The question is how
we plan to support it in Bigtop.

I can think of several items to discuss for this feature:
1. Docker images name conventions:
For this, I'd suggest to using this:
* for actual underlying docker image running on specific platform, name
it as:
bigtop/:---,
say: bigtop/puppet:trunk-debian-9-amd64,
bigtop/slaves:1.3.0-fedora-26-aarch64
* for virtual image, name it as: bigtop/:
--,
say:  bigtop/puppet:trunk-debian-9,  bigtop/slaves:1.3.0-fedora-26
2. How to create these images/manifests
I didn't find any cmds/scripts in Bigtop source to do the docker hub
uploading work. So I guess we're doing it manually for now. If we're going
to use multi-arch, we need to upload the virtual image's manifest file to
docker hub besides those arch dependent images.
Docker cli has a cmd to do such work: docker manifest. But it is not in
distros' docker releases. And it is marked as "experimental" feature, user
needs to change docker settings to use it. See
https://docs.docker.com/edge/engine/reference/commandline/manifest/#description.
I created a shell script version (
https://github.com/JunHe77/docker-manifest-sh) to do the same thing, use
same cmd syntax. Hope this can help.
3. Update existing settings/yaml files to use the virutal distro image.
Once the images/manifests are ready in docker hub, we can update
existing settings/yaml files to use "virtual" bigtop images. This should
take too much efforts. And we'll get unified, beautiful code for different
archs.

Pls raise your thoughts, questions, and etc. Once we settle these
workflows/specs down, I'm sure it will be ready before Sept.

Thanks.

2018-05-28 1:45 GMT+08:00 Evans Ye :

> I'd say this is a great idea.
> The only reason we haven't adopt multi-arch is because we started to adopt
> docker at early stage so there's no such feature then.
>
> Jun can you roughly estimate the effort? If we're going for it, we better
> have it ready in 1.3.0 before ApacheCon which is around September.
>
>
> 2018-05-17 17:19 GMT+08:00 Naresh Bhat :
>
> > On 17 May 2018 at 13:21, Jun HE  wrote:
> > > Hi, guys,
> > >
> > > As bigtop has simplified the way to create docker images for distros,
> by
> > > BIGTOP-2922, I'm thinking of multi-arch docker images support to ease
> > > building/test/development.
> > >
> > > Multi-arch feature has been supported since 2017, you may check this
> > link:
> > > https://blog.docker.com/2017/11/multi-arch-all-the-things/
> > > In simple words, multi-arch is a virtual image which contains several
> > > architecture dependent images. When user try to pull this virtual image
> > > dockhub will send back the actual image based on the architecture he is
> > > using.
> > >
> > > Use multi-arch will help:
> > > * development: user don't have to issue different cmd to pull
> > puppet/slaves
> > > images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of
> > "docker
> > > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull
> > > bigtop/slaves:trunk-debian-9-aarch64"
> > > * test: unified yaml file for same distro. refer example in
> > > provisioner/docker/config_*
> > >
> > > So how do you think? :)
> >
> > Thanks Jun for initiating the thread.
> > +1 this makes sense. Is there any reason why we have not consider for
> > using multi-arch images ? If anybody explains advantage/disadvantage
> > of using multi-arch docker image in bigtop, it will help us to
> > understand in a better way.
> >
> > >
> > > Regards,
> > >
> > > Jun
> >
>

Re: Create multiarch docker images for bigtop-puppet/slaves

[Evans Ye]

I'd say this is a great idea.
The only reason we haven't adopt multi-arch is because we started to adopt
docker at early stage so there's no such feature then.

Jun can you roughly estimate the effort? If we're going for it, we better
have it ready in 1.3.0 before ApacheCon which is around September.


2018-05-17 17:19 GMT+08:00 Naresh Bhat :

> On 17 May 2018 at 13:21, Jun HE  wrote:
> > Hi, guys,
> >
> > As bigtop has simplified the way to create docker images for distros, by
> > BIGTOP-2922, I'm thinking of multi-arch docker images support to ease
> > building/test/development.
> >
> > Multi-arch feature has been supported since 2017, you may check this
> link:
> > https://blog.docker.com/2017/11/multi-arch-all-the-things/
> > In simple words, multi-arch is a virtual image which contains several
> > architecture dependent images. When user try to pull this virtual image
> > dockhub will send back the actual image based on the architecture he is
> > using.
> >
> > Use multi-arch will help:
> > * development: user don't have to issue different cmd to pull
> puppet/slaves
> > images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of
> "docker
> > pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull
> > bigtop/slaves:trunk-debian-9-aarch64"
> > * test: unified yaml file for same distro. refer example in
> > provisioner/docker/config_*
> >
> > So how do you think? :)
>
> Thanks Jun for initiating the thread.
> +1 this makes sense. Is there any reason why we have not consider for
> using multi-arch images ? If anybody explains advantage/disadvantage
> of using multi-arch docker image in bigtop, it will help us to
> understand in a better way.
>
> >
> > Regards,
> >
> > Jun
>

Re: Create multiarch docker images for bigtop-puppet/slaves

[Naresh Bhat]

On 17 May 2018 at 13:21, Jun HE  wrote:
> Hi, guys,
>
> As bigtop has simplified the way to create docker images for distros, by
> BIGTOP-2922, I'm thinking of multi-arch docker images support to ease
> building/test/development.
>
> Multi-arch feature has been supported since 2017, you may check this link:
> https://blog.docker.com/2017/11/multi-arch-all-the-things/
> In simple words, multi-arch is a virtual image which contains several
> architecture dependent images. When user try to pull this virtual image
> dockhub will send back the actual image based on the architecture he is
> using.
>
> Use multi-arch will help:
> * development: user don't have to issue different cmd to pull puppet/slaves
> images, "docker pull bigtop/slaves:trunk-debian-9" is enough instead of 
> "docker
> pull bigtop/slaves:trunk-debian-9-ppc64le" or "docker pull
> bigtop/slaves:trunk-debian-9-aarch64"
> * test: unified yaml file for same distro. refer example in
> provisioner/docker/config_*
>
> So how do you think? :)

Thanks Jun for initiating the thread.
+1 this makes sense. Is there any reason why we have not consider for
using multi-arch images ? If anybody explains advantage/disadvantage
of using multi-arch docker image in bigtop, it will help us to
understand in a better way.

>
> Regards,
>
> Jun