Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Hongbin Lu]

Another use case I can think of is to cache the required docker images in the 
Glance image.

This is an important use case because we have containerized most of the COE 
components (e.g. kube-scheduler, swarm-manager, etc.). As a result, each bay 
needs to pull docker images over the Internet on provisioning or scaling stage. 
If a large number of bays pull docker images at the same time, it will generate 
a lot of traffic. Therefore, it is desirable to have all the required docker 
images pre-downloaded into the Glance image. I expect we can leverage 
diskimage-builder to achieve the goal.

Best regards,
Hongbin

From: Ton Ngo [mailto:t...@us.ibm.com]
Sent: March-29-16 4:54 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [magnum] Generate atomic images using 
diskimage-builder


In multiple occasions in the past, we have had to use version of some software 
that's not available yet
in the upstream image for bug fixes or new features (Kubernetes, Docker, 
Flannel,...). Eventually the upstream
image would catch up, but having the tool to customize let us push forward with 
development, and gate tests
if it makes sense.

Ton Ngo,


[Inactive hide details for Yolanda Robla Mota ---03/29/2016 01:35:48 PM---So 
the advantages I can see with diskimage-builder are]Yolanda Robla Mota 
---03/29/2016 01:35:48 PM---So the advantages I can see with diskimage-builder 
are: - we reuse the same tooling that is present

From: Yolanda Robla Mota 
mailto:yolanda.robla-m...@hpe.com>>
To: 
mailto:openstack-dev@lists.openstack.org>>
Date: 03/29/2016 01:35 PM
Subject: Re: [openstack-dev] [magnum] Generate atomic images using 
diskimage-builder





So the advantages I can see with diskimage-builder are:
- we reuse the same tooling that is present in other openstack projects
to generate images, rather than relying on an external image
- it improves the control we have on the contents of the image, instead
of seeing that as a black box. At the moment we can rely on the default
tree for fedora 23, but this can be updated per magnum needs
- reusability: we have atomic 23 now, but why not create magnum images
with dib, for ubuntu, or any other distros ? Relying on
diskimage-builder makes it easy and flexible, because it's a matter of
adding the right elements.

Best
Yolanda

El 29/03/16 a las 21:54, Steven Dake (stdake) escribió:
> Adrian,
>
> Makes sense.  Do the images have to be built to be mirrored though?  Can't
> they just be put on the mirror sites fro upstream?
>
> Thanks
> -steve
>
> On 3/29/16, 11:02 AM, "Adrian Otto" 
> mailto:adrian.o...@rackspace.com>> wrote:
>
>> Steve,
>>
>> I¹m very interested in having an image locally cached in glance in each
>> of the clouds used by OpenStack infra. The local caching of the glance
>> images will produce much faster gate testing times. I don¹t care about
>> how the images are built, but we really do care about the performance
>> outcome.
>>
>> Adrian
>>
>>> On Mar 29, 2016, at 10:38 AM, Steven Dake (stdake) 
>>> mailto:std...@cisco.com>>
>>> wrote:
>>>
>>> Yolanda,
>>>
>>> That is a fantastic objective.  Matthieu asked why build our own images
>>> if
>>> the upstream images work and need no further customization?
>>>
>>> Regards
>>> -steve
>>>
>>> On 3/29/16, 1:57 AM, "Yolanda Robla Mota" 
>>> mailto:yolanda.robla-m...@hpe.com>>
>>> wrote:
>>>
>>>> Hi
>>>> The idea is to build own images using diskimage-builder, rather than
>>>> downloading the image from external sources. By that way, the image can
>>>> live in our mirrors, and is built using the same pattern as other
>>>> images
>>>> used in OpenStack.
>>>> It also opens the door to customize the images, using custom trees, if
>>>> there is a need for it. Actually we rely on official tree for Fedora 23
>>>> Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as
>>>> default.
>>>>
>>>> Best,
>>>> Yolanda
>>>>
>>>> El 29/03/16 a las 10:17, Mathieu Velten escribió:
>>>>> Hi,
>>>>>
>>>>> We are using the official Fedora Atomic 23 images here (on Mitaka M1
>>>>> however) and it seems to work fine with at least Kubernetes and Docker
>>>>> Swarm.
>>>>> Any reason to continue building specific Magnum image ?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Mathieu
>>>>>
>>>>> Le mercredi 2

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Ton Ngo]

In multiple occasions in the past, we have had to use version of some
software that's not available yet
in the upstream image for bug fixes or new features (Kubernetes, Docker,
Flannel,...).  Eventually the upstream
image would catch up, but having the tool to customize let us push forward
with development, and gate tests
if it makes sense.

Ton Ngo,




From:   Yolanda Robla Mota 
To: 
Date:   03/29/2016 01:35 PM
Subject:Re: [openstack-dev] [magnum] Generate atomic images using
    diskimage-builder



So the advantages I can see with diskimage-builder are:
- we reuse the same tooling that is present in other openstack projects
to generate images, rather than relying on an external image
- it improves the control we have on the contents of the image, instead
of seeing that as a black box. At the moment we can rely on the default
tree for fedora 23, but this can be updated per magnum needs
- reusability: we have atomic 23 now, but why not create magnum images
with dib, for ubuntu, or any other distros ? Relying on
diskimage-builder makes it easy and flexible, because it's a matter of
adding the right elements.

Best
Yolanda

El 29/03/16 a las 21:54, Steven Dake (stdake) escribió:
> Adrian,
>
> Makes sense.  Do the images have to be built to be mirrored though?
Can't
> they just be put on the mirror sites fro upstream?
>
> Thanks
> -steve
>
> On 3/29/16, 11:02 AM, "Adrian Otto"  wrote:
>
>> Steve,
>>
>> I¹m very interested in having an image locally cached in glance in each
>> of the clouds used by OpenStack infra. The local caching of the glance
>> images will produce much faster gate testing times. I don¹t care about
>> how the images are built, but we really do care about the performance
>> outcome.
>>
>> Adrian
>>
>>> On Mar 29, 2016, at 10:38 AM, Steven Dake (stdake) 
>>> wrote:
>>>
>>> Yolanda,
>>>
>>> That is a fantastic objective.  Matthieu asked why build our own images
>>> if
>>> the upstream images work and need no further customization?
>>>
>>> Regards
>>> -steve
>>>
>>> On 3/29/16, 1:57 AM, "Yolanda Robla Mota" 
>>> wrote:
>>>
>>>> Hi
>>>> The idea is to build own images using diskimage-builder, rather than
>>>> downloading the image from external sources. By that way, the image
can
>>>> live in our mirrors, and is built using the same pattern as other
>>>> images
>>>> used in OpenStack.
>>>> It also opens the door to customize the images, using custom trees, if
>>>> there is a need for it. Actually we rely on official tree for Fedora
23
>>>> Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as
>>>> default.
>>>>
>>>> Best,
>>>> Yolanda
>>>>
>>>> El 29/03/16 a las 10:17, Mathieu Velten escribió:
>>>>> Hi,
>>>>>
>>>>> We are using the official Fedora Atomic 23 images here (on Mitaka M1
>>>>> however) and it seems to work fine with at least Kubernetes and
Docker
>>>>> Swarm.
>>>>> Any reason to continue building specific Magnum image ?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Mathieu
>>>>>
>>>>> Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :
>>>>>> Hi
>>>>>> I wanted to start a discussion on how Fedora Atomic images are being
>>>>>> built. Currently the process for generating the atomic images used
>>>>>> on
>>>>>> Magnum is described here:
>>>>>>
http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
>>>>>> l.
>>>>>> The image needs to be built manually, uploaded to fedorapeople, and
>>>>>> then
>>>>>> consumed from there in the magnum tests.
>>>>>> I have been working on a feature to allow diskimage-builder to
>>>>>> generate
>>>>>> these images. The code that makes it possible is here:
>>>>>> https://review.openstack.org/287167
>>>>>> This will allow that magnum images are generated on infra, using
>>>>>> diskimage-builder element. This element also has the ability to
>>>>>> consume
>>>>>> any tree we need, so images can be customized on demand. I generated
>>>>>> one
>>>>>> image using this element, and uploaded to fedora people. The image
>>>>&g

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Yolanda Robla Mota]

So the advantages I can see with diskimage-builder are:
- we reuse the same tooling that is present in other openstack projects 
to generate images, rather than relying on an external image
- it improves the control we have on the contents of the image, instead 
of seeing that as a black box. At the moment we can rely on the default 
tree for fedora 23, but this can be updated per magnum needs
- reusability: we have atomic 23 now, but why not create magnum images 
with dib, for ubuntu, or any other distros ? Relying on 
diskimage-builder makes it easy and flexible, because it's a matter of 
adding the right elements.


Best
Yolanda

El 29/03/16 a las 21:54, Steven Dake (stdake) escribió:

Adrian,

Makes sense.  Do the images have to be built to be mirrored though?  Can't
they just be put on the mirror sites fro upstream?

Thanks
-steve

On 3/29/16, 11:02 AM, "Adrian Otto"  wrote:


Steve,

I¹m very interested in having an image locally cached in glance in each
of the clouds used by OpenStack infra. The local caching of the glance
images will produce much faster gate testing times. I don¹t care about
how the images are built, but we really do care about the performance
outcome.

Adrian


On Mar 29, 2016, at 10:38 AM, Steven Dake (stdake) 
wrote:

Yolanda,

That is a fantastic objective.  Matthieu asked why build our own images
if
the upstream images work and need no further customization?

Regards
-steve

On 3/29/16, 1:57 AM, "Yolanda Robla Mota" 
wrote:


Hi
The idea is to build own images using diskimage-builder, rather than
downloading the image from external sources. By that way, the image can
live in our mirrors, and is built using the same pattern as other
images
used in OpenStack.
It also opens the door to customize the images, using custom trees, if
there is a need for it. Actually we rely on official tree for Fedora 23
Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as
default.

Best,
Yolanda

El 29/03/16 a las 10:17, Mathieu Velten escribió:

Hi,

We are using the official Fedora Atomic 23 images here (on Mitaka M1
however) and it seems to work fine with at least Kubernetes and Docker
Swarm.
Any reason to continue building specific Magnum image ?

Regards,

Mathieu

Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :

Hi
I wanted to start a discussion on how Fedora Atomic images are being
built. Currently the process for generating the atomic images used
on
Magnum is described here:
http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
l.
The image needs to be built manually, uploaded to fedorapeople, and
then
consumed from there in the magnum tests.
I have been working on a feature to allow diskimage-builder to
generate
these images. The code that makes it possible is here:
https://review.openstack.org/287167
This will allow that magnum images are generated on infra, using
diskimage-builder element. This element also has the ability to
consume
any tree we need, so images can be customized on demand. I generated
one
image using this element, and uploaded to fedora people. The image
has
passed tests, and has been validated by several people.

So i'm raising that topic to decide what should be the next steps.
This
change to generate fedora-atomic images has not already landed into
diskimage-builder. But we have two options here:
- add this element to generic diskimage-builder elements, as i'm
doing now
- generate this element internally on magnum. So we can have a
directory
in magnum project, called "elements", and have the fedora-atomic
element
here. This will give us more control on the element behaviour, and
will
allow to update the element without waiting for external reviews.

Once the code for diskimage-builder has landed, another step can be
to
periodically generate images using a magnum job, and upload these
images
to OpenStack Infra mirrors. Currently the image is based on Fedora
F23,
docker-host tree. But different images can be generated if we need a
better option.

As soon as the images are available on internal infra mirrors, the
tests
can be changed, to consume these internals images. By this way the
tests
can be a bit faster (i know that the bottleneck is on the functional
testing, but if we reduce the download time it can help), and tests
can
be more reilable, because we will be removing an external dependency.

So i'd like to get more feedback on this topic, options and next
steps
to achieve the goals. Best



___
__
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Yolanda Robla Mota
Cloud Automation and Distribution Engineer
+34 605641639
yolanda.robla-m...@hpe.com




__
OpenStack Development Mailing List (not for usage questions)
Unsub

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Adrian Otto]

Steve,

I will defer to the experts in openstack-infra on this one. As long as the 
image works without modifications, then I think it would be fine to cache the 
upstream one. Practically speaking, I do anticipate a point at which we will 
want to adjust something in the image, and it will be nice to have a well 
defined point of customization in place for that in advance.

Adrian

On Mar 29, 2016, at 12:54 PM, Steven Dake (stdake) 
mailto:std...@cisco.com>> wrote:

Adrian,

Makes sense.  Do the images have to be built to be mirrored though?  Can't
they just be put on the mirror sites fro upstream?

Thanks
-steve

On 3/29/16, 11:02 AM, "Adrian Otto" 
mailto:adrian.o...@rackspace.com>> wrote:

Steve,

I¹m very interested in having an image locally cached in glance in each
of the clouds used by OpenStack infra. The local caching of the glance
images will produce much faster gate testing times. I don¹t care about
how the images are built, but we really do care about the performance
outcome.

Adrian

On Mar 29, 2016, at 10:38 AM, Steven Dake (stdake) 
mailto:std...@cisco.com>>
wrote:

Yolanda,

That is a fantastic objective.  Matthieu asked why build our own images
if
the upstream images work and need no further customization?

Regards
-steve

On 3/29/16, 1:57 AM, "Yolanda Robla Mota" 
mailto:yolanda.robla-m...@hpe.com>>
wrote:

Hi
The idea is to build own images using diskimage-builder, rather than
downloading the image from external sources. By that way, the image can
live in our mirrors, and is built using the same pattern as other
images
used in OpenStack.
It also opens the door to customize the images, using custom trees, if
there is a need for it. Actually we rely on official tree for Fedora 23
Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as
default.

Best,
Yolanda

El 29/03/16 a las 10:17, Mathieu Velten escribió:
Hi,

We are using the official Fedora Atomic 23 images here (on Mitaka M1
however) and it seems to work fine with at least Kubernetes and Docker
Swarm.
Any reason to continue building specific Magnum image ?

Regards,

Mathieu

Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :
Hi
I wanted to start a discussion on how Fedora Atomic images are being
built. Currently the process for generating the atomic images used
on
Magnum is described here:
http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
l.
The image needs to be built manually, uploaded to fedorapeople, and
then
consumed from there in the magnum tests.
I have been working on a feature to allow diskimage-builder to
generate
these images. The code that makes it possible is here:
https://review.openstack.org/287167
This will allow that magnum images are generated on infra, using
diskimage-builder element. This element also has the ability to
consume
any tree we need, so images can be customized on demand. I generated
one
image using this element, and uploaded to fedora people. The image
has
passed tests, and has been validated by several people.

So i'm raising that topic to decide what should be the next steps.
This
change to generate fedora-atomic images has not already landed into
diskimage-builder. But we have two options here:
- add this element to generic diskimage-builder elements, as i'm
doing now
- generate this element internally on magnum. So we can have a
directory
in magnum project, called "elements", and have the fedora-atomic
element
here. This will give us more control on the element behaviour, and
will
allow to update the element without waiting for external reviews.

Once the code for diskimage-builder has landed, another step can be
to
periodically generate images using a magnum job, and upload these
images
to OpenStack Infra mirrors. Currently the image is based on Fedora
F23,
docker-host tree. But different images can be generated if we need a
better option.

As soon as the images are available on internal infra mirrors, the
tests
can be changed, to consume these internals images. By this way the
tests
can be a bit faster (i know that the bottleneck is on the functional
testing, but if we reduce the download time it can help), and tests
can
be more reilable, because we will be removing an external dependency.

So i'd like to get more feedback on this topic, options and next
steps
to achieve the goals. Best



___
__
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Yolanda Robla Mota
Cloud Automation and Distribution Engineer
+34 605641639
yolanda.robla-m...@hpe.com




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Steven Dake (stdake)]

Adrian,

Makes sense.  Do the images have to be built to be mirrored though?  Can't
they just be put on the mirror sites fro upstream?

Thanks
-steve

On 3/29/16, 11:02 AM, "Adrian Otto"  wrote:

>Steve,
>
>I¹m very interested in having an image locally cached in glance in each
>of the clouds used by OpenStack infra. The local caching of the glance
>images will produce much faster gate testing times. I don¹t care about
>how the images are built, but we really do care about the performance
>outcome.
>
>Adrian
>
>> On Mar 29, 2016, at 10:38 AM, Steven Dake (stdake) 
>>wrote:
>> 
>> Yolanda,
>> 
>> That is a fantastic objective.  Matthieu asked why build our own images
>>if
>> the upstream images work and need no further customization?
>> 
>> Regards
>> -steve
>> 
>> On 3/29/16, 1:57 AM, "Yolanda Robla Mota" 
>> wrote:
>> 
>>> Hi
>>> The idea is to build own images using diskimage-builder, rather than
>>> downloading the image from external sources. By that way, the image can
>>> live in our mirrors, and is built using the same pattern as other
>>>images
>>> used in OpenStack.
>>> It also opens the door to customize the images, using custom trees, if
>>> there is a need for it. Actually we rely on official tree for Fedora 23
>>> Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as
>>> default.
>>> 
>>> Best,
>>> Yolanda
>>> 
>>> El 29/03/16 a las 10:17, Mathieu Velten escribió:
 Hi,
 
 We are using the official Fedora Atomic 23 images here (on Mitaka M1
 however) and it seems to work fine with at least Kubernetes and Docker
 Swarm.
 Any reason to continue building specific Magnum image ?
 
 Regards,
 
 Mathieu
 
 Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :
> Hi
> I wanted to start a discussion on how Fedora Atomic images are being
> built. Currently the process for generating the atomic images used
> on
> Magnum is described here:
> http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
> l.
> The image needs to be built manually, uploaded to fedorapeople, and
> then
> consumed from there in the magnum tests.
> I have been working on a feature to allow diskimage-builder to
> generate
> these images. The code that makes it possible is here:
> https://review.openstack.org/287167
> This will allow that magnum images are generated on infra, using
> diskimage-builder element. This element also has the ability to
> consume
> any tree we need, so images can be customized on demand. I generated
> one
> image using this element, and uploaded to fedora people. The image
> has
> passed tests, and has been validated by several people.
> 
> So i'm raising that topic to decide what should be the next steps.
> This
> change to generate fedora-atomic images has not already landed into
> diskimage-builder. But we have two options here:
> - add this element to generic diskimage-builder elements, as i'm
> doing now
> - generate this element internally on magnum. So we can have a
> directory
> in magnum project, called "elements", and have the fedora-atomic
> element
> here. This will give us more control on the element behaviour, and
> will
> allow to update the element without waiting for external reviews.
> 
> Once the code for diskimage-builder has landed, another step can be
> to
> periodically generate images using a magnum job, and upload these
> images
> to OpenStack Infra mirrors. Currently the image is based on Fedora
> F23,
> docker-host tree. But different images can be generated if we need a
> better option.
> 
> As soon as the images are available on internal infra mirrors, the
> tests
> can be changed, to consume these internals images. By this way the
> tests
> can be a bit faster (i know that the bottleneck is on the functional
> testing, but if we reduce the download time it can help), and tests
> can
> be more reilable, because we will be removing an external dependency.
> 
> So i'd like to get more feedback on this topic, options and next
> steps
> to achieve the goals. Best
> 
 
 
___
__
 _
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: 
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> -- 
>>> Yolanda Robla Mota
>>> Cloud Automation and Distribution Engineer
>>> +34 605641639
>>> yolanda.robla-m...@hpe.com
>>> 
>>> 
>>> 
>>>
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http:

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Adrian Otto]

Steve,

I’m very interested in having an image locally cached in glance in each of the 
clouds used by OpenStack infra. The local caching of the glance images will 
produce much faster gate testing times. I don’t care about how the images are 
built, but we really do care about the performance outcome.

Adrian

> On Mar 29, 2016, at 10:38 AM, Steven Dake (stdake)  wrote:
> 
> Yolanda,
> 
> That is a fantastic objective.  Matthieu asked why build our own images if
> the upstream images work and need no further customization?
> 
> Regards
> -steve
> 
> On 3/29/16, 1:57 AM, "Yolanda Robla Mota" 
> wrote:
> 
>> Hi
>> The idea is to build own images using diskimage-builder, rather than
>> downloading the image from external sources. By that way, the image can
>> live in our mirrors, and is built using the same pattern as other images
>> used in OpenStack.
>> It also opens the door to customize the images, using custom trees, if
>> there is a need for it. Actually we rely on official tree for Fedora 23
>> Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as
>> default.
>> 
>> Best,
>> Yolanda
>> 
>> El 29/03/16 a las 10:17, Mathieu Velten escribió:
>>> Hi,
>>> 
>>> We are using the official Fedora Atomic 23 images here (on Mitaka M1
>>> however) and it seems to work fine with at least Kubernetes and Docker
>>> Swarm.
>>> Any reason to continue building specific Magnum image ?
>>> 
>>> Regards,
>>> 
>>> Mathieu
>>> 
>>> Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :
 Hi
 I wanted to start a discussion on how Fedora Atomic images are being
 built. Currently the process for generating the atomic images used
 on
 Magnum is described here:
 http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
 l.
 The image needs to be built manually, uploaded to fedorapeople, and
 then
 consumed from there in the magnum tests.
 I have been working on a feature to allow diskimage-builder to
 generate
 these images. The code that makes it possible is here:
 https://review.openstack.org/287167
 This will allow that magnum images are generated on infra, using
 diskimage-builder element. This element also has the ability to
 consume
 any tree we need, so images can be customized on demand. I generated
 one
 image using this element, and uploaded to fedora people. The image
 has
 passed tests, and has been validated by several people.
 
 So i'm raising that topic to decide what should be the next steps.
 This
 change to generate fedora-atomic images has not already landed into
 diskimage-builder. But we have two options here:
 - add this element to generic diskimage-builder elements, as i'm
 doing now
 - generate this element internally on magnum. So we can have a
 directory
 in magnum project, called "elements", and have the fedora-atomic
 element
 here. This will give us more control on the element behaviour, and
 will
 allow to update the element without waiting for external reviews.
 
 Once the code for diskimage-builder has landed, another step can be
 to
 periodically generate images using a magnum job, and upload these
 images
 to OpenStack Infra mirrors. Currently the image is based on Fedora
 F23,
 docker-host tree. But different images can be generated if we need a
 better option.
 
 As soon as the images are available on internal infra mirrors, the
 tests
 can be changed, to consume these internals images. By this way the
 tests
 can be a bit faster (i know that the bottleneck is on the functional
 testing, but if we reduce the download time it can help), and tests
 can
 be more reilable, because we will be removing an external dependency.
 
 So i'd like to get more feedback on this topic, options and next
 steps
 to achieve the goals. Best
 
>>> 
>>> _
>>> _
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> -- 
>> Yolanda Robla Mota
>> Cloud Automation and Distribution Engineer
>> +34 605641639
>> yolanda.robla-m...@hpe.com
>> 
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Steven Dake (stdake)]

Yolanda,

That is a fantastic objective.  Matthieu asked why build our own images if
the upstream images work and need no further customization?

Regards
-steve

On 3/29/16, 1:57 AM, "Yolanda Robla Mota" 
wrote:

>Hi
>The idea is to build own images using diskimage-builder, rather than
>downloading the image from external sources. By that way, the image can
>live in our mirrors, and is built using the same pattern as other images
>used in OpenStack.
>It also opens the door to customize the images, using custom trees, if
>there is a need for it. Actually we rely on official tree for Fedora 23
>Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as
>default.
>
>Best,
>Yolanda
>
>El 29/03/16 a las 10:17, Mathieu Velten escribió:
>> Hi,
>>
>> We are using the official Fedora Atomic 23 images here (on Mitaka M1
>> however) and it seems to work fine with at least Kubernetes and Docker
>> Swarm.
>> Any reason to continue building specific Magnum image ?
>>
>> Regards,
>>
>> Mathieu
>>
>> Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :
>>> Hi
>>> I wanted to start a discussion on how Fedora Atomic images are being
>>> built. Currently the process for generating the atomic images used
>>> on
>>> Magnum is described here:
>>> http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
>>> l.
>>> The image needs to be built manually, uploaded to fedorapeople, and
>>> then
>>> consumed from there in the magnum tests.
>>> I have been working on a feature to allow diskimage-builder to
>>> generate
>>> these images. The code that makes it possible is here:
>>> https://review.openstack.org/287167
>>> This will allow that magnum images are generated on infra, using
>>> diskimage-builder element. This element also has the ability to
>>> consume
>>> any tree we need, so images can be customized on demand. I generated
>>> one
>>> image using this element, and uploaded to fedora people. The image
>>> has
>>> passed tests, and has been validated by several people.
>>>
>>> So i'm raising that topic to decide what should be the next steps.
>>> This
>>> change to generate fedora-atomic images has not already landed into
>>> diskimage-builder. But we have two options here:
>>> - add this element to generic diskimage-builder elements, as i'm
>>> doing now
>>> - generate this element internally on magnum. So we can have a
>>> directory
>>> in magnum project, called "elements", and have the fedora-atomic
>>> element
>>> here. This will give us more control on the element behaviour, and
>>> will
>>> allow to update the element without waiting for external reviews.
>>>
>>> Once the code for diskimage-builder has landed, another step can be
>>> to
>>> periodically generate images using a magnum job, and upload these
>>> images
>>> to OpenStack Infra mirrors. Currently the image is based on Fedora
>>> F23,
>>> docker-host tree. But different images can be generated if we need a
>>> better option.
>>>
>>> As soon as the images are available on internal infra mirrors, the
>>> tests
>>> can be changed, to consume these internals images. By this way the
>>> tests
>>> can be a bit faster (i know that the bottleneck is on the functional
>>> testing, but if we reduce the download time it can help), and tests
>>> can
>>> be more reilable, because we will be removing an external dependency.
>>>
>>> So i'd like to get more feedback on this topic, options and next
>>> steps
>>> to achieve the goals. Best
>>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>-- 
>Yolanda Robla Mota
>Cloud Automation and Distribution Engineer
>+34 605641639
>yolanda.robla-m...@hpe.com
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Yolanda Robla Mota]

Hi
The idea is to build own images using diskimage-builder, rather than 
downloading the image from external sources. By that way, the image can 
live in our mirrors, and is built using the same pattern as other images 
used in OpenStack.
It also opens the door to customize the images, using custom trees, if 
there is a need for it. Actually we rely on official tree for Fedora 23 
Atomic (https://dl.fedoraproject.org/pub/fedora/linux/atomic/23/) as 
default.


Best,
Yolanda

El 29/03/16 a las 10:17, Mathieu Velten escribió:

Hi,

We are using the official Fedora Atomic 23 images here (on Mitaka M1
however) and it seems to work fine with at least Kubernetes and Docker
Swarm.
Any reason to continue building specific Magnum image ?

Regards,

Mathieu

Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :

Hi
I wanted to start a discussion on how Fedora Atomic images are being
built. Currently the process for generating the atomic images used
on
Magnum is described here:
http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
l.
The image needs to be built manually, uploaded to fedorapeople, and
then
consumed from there in the magnum tests.
I have been working on a feature to allow diskimage-builder to
generate
these images. The code that makes it possible is here:
https://review.openstack.org/287167
This will allow that magnum images are generated on infra, using
diskimage-builder element. This element also has the ability to
consume
any tree we need, so images can be customized on demand. I generated
one
image using this element, and uploaded to fedora people. The image
has
passed tests, and has been validated by several people.

So i'm raising that topic to decide what should be the next steps.
This
change to generate fedora-atomic images has not already landed into
diskimage-builder. But we have two options here:
- add this element to generic diskimage-builder elements, as i'm
doing now
- generate this element internally on magnum. So we can have a
directory
in magnum project, called "elements", and have the fedora-atomic
element
here. This will give us more control on the element behaviour, and
will
allow to update the element without waiting for external reviews.

Once the code for diskimage-builder has landed, another step can be
to
periodically generate images using a magnum job, and upload these
images
to OpenStack Infra mirrors. Currently the image is based on Fedora
F23,
docker-host tree. But different images can be generated if we need a
better option.

As soon as the images are available on internal infra mirrors, the
tests
can be changed, to consume these internals images. By this way the
tests
can be a bit faster (i know that the bottleneck is on the functional
testing, but if we reduce the download time it can help), and tests
can
be more reilable, because we will be removing an external dependency.

So i'd like to get more feedback on this topic, options and next
steps
to achieve the goals. Best


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
Yolanda Robla Mota
Cloud Automation and Distribution Engineer
+34 605641639
yolanda.robla-m...@hpe.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Mathieu Velten]

Hi,

We are using the official Fedora Atomic 23 images here (on Mitaka M1
however) and it seems to work fine with at least Kubernetes and Docker
Swarm.
Any reason to continue building specific Magnum image ?

Regards,

Mathieu

Le mercredi 23 mars 2016 à 12:09 +0100, Yolanda Robla Mota a écrit :
> Hi
> I wanted to start a discussion on how Fedora Atomic images are being 
> built. Currently the process for generating the atomic images used
> on 
> Magnum is described here: 
> http://docs.openstack.org/developer/magnum/dev/build-atomic-image.htm
> l.
> The image needs to be built manually, uploaded to fedorapeople, and
> then 
> consumed from there in the magnum tests.
> I have been working on a feature to allow diskimage-builder to
> generate 
> these images. The code that makes it possible is here: 
> https://review.openstack.org/287167
> This will allow that magnum images are generated on infra, using 
> diskimage-builder element. This element also has the ability to
> consume 
> any tree we need, so images can be customized on demand. I generated
> one 
> image using this element, and uploaded to fedora people. The image
> has 
> passed tests, and has been validated by several people.
> 
> So i'm raising that topic to decide what should be the next steps.
> This 
> change to generate fedora-atomic images has not already landed into 
> diskimage-builder. But we have two options here:
> - add this element to generic diskimage-builder elements, as i'm
> doing now
> - generate this element internally on magnum. So we can have a
> directory 
> in magnum project, called "elements", and have the fedora-atomic
> element 
> here. This will give us more control on the element behaviour, and
> will 
> allow to update the element without waiting for external reviews.
> 
> Once the code for diskimage-builder has landed, another step can be
> to 
> periodically generate images using a magnum job, and upload these
> images 
> to OpenStack Infra mirrors. Currently the image is based on Fedora
> F23, 
> docker-host tree. But different images can be generated if we need a 
> better option.
> 
> As soon as the images are available on internal infra mirrors, the
> tests 
> can be changed, to consume these internals images. By this way the
> tests 
> can be a bit faster (i know that the bottleneck is on the functional 
> testing, but if we reduce the download time it can help), and tests
> can 
> be more reilable, because we will be removing an external dependency.
> 
> So i'd like to get more feedback on this topic, options and next
> steps 
> to achieve the goals. Best
> 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Yolanda Robla Mota]

Hi
I have also been talking with diskimage-builder cores and they suggested 
we better start with the element on magnum. So I moved the change here:

https://review.openstack.org/#/c/296719/

About diskimage-size, I'll dig a bit more on it. It can be for two causes:
1. To generate the fedora-atomic image, I first install a fedora-minimal 
image, and install ostree packages and the dependencies there. Based on 
that, i execute the ostree commands, and then I update the grub config 
to boot with the new image. But the old contents are still there, so 
this is potentially increasing the size. I will take a look at possible 
cleanup solutions.
2. The way diskimage-builder compresses the image. It can be different 
from the one that Fedora is using, and less effective. I saw discussions 
in diskimage-builder about that, and there have been recent improvements 
as https://review.openstack.org/290944 . Is worth investigating a bit 
more as well.


Best
Yolanda

El 23/03/16 a las 18:10, Ton Ngo escribió:


Hi Yolanda,
Thank you for making a huge improvement from the manual process of 
building the Fedora Atomic image.
Although Atomic does publish a public OpenStack image that is being 
considered in this patch:

https://review.openstack.org/#/c/276232/
in the past we have come to many situations where we need an image 
with specific version of certain software
for features or bug fixes (Kubernetes, Docker, Flannel, ...). So the 
automated and customizable build process

will be very helpful.

With respect to where to land the patch, I think diskimage-builder is 
a reasonable target.
If it does not land there, Magnum does currently have 2 sets of 
diskimage-builder elements for Mesos image
and Ironic image, so it is also reasonable to submit the patch to 
Magnum. With the new push to reorganize
into drivers for COE and distro, the elements would be a natural fit 
for Fedora Atomic.


As for periodic image build, it's a good idea to stay current with the 
distro, but we should avoid the situation
where something new in the image breaks a COE and we are stuck for 
awhile until a fix is made. So instead of
an automated periodic build, we might want to stage the new image to 
make sure it's good before switching.


One question: I notice the image built by DIB is 871MB, similar to the 
manually built image, while the
public image from Atomic is 486MB. It might be worthwhile to 
understand the difference.


Ton Ngo,

Inactive hide details for Yolanda Robla Mota ---03/23/2016 04:12:54 
AM---Hi I wanted to start a discussion on how Fedora AtomicYolanda 
Robla Mota ---03/23/2016 04:12:54 AM---Hi I wanted to start a 
discussion on how Fedora Atomic images are being


From: Yolanda Robla Mota 
To: 
Date: 03/23/2016 04:12 AM
Subject: [openstack-dev] [magnum] Generate atomic images using 
diskimage-builder






Hi
I wanted to start a discussion on how Fedora Atomic images are being
built. Currently the process for generating the atomic images used on
Magnum is described here:
http://docs.openstack.org/developer/magnum/dev/build-atomic-image.html.
The image needs to be built manually, uploaded to fedorapeople, and then
consumed from there in the magnum tests.
I have been working on a feature to allow diskimage-builder to generate
these images. The code that makes it possible is here:
https://review.openstack.org/287167
This will allow that magnum images are generated on infra, using
diskimage-builder element. This element also has the ability to consume
any tree we need, so images can be customized on demand. I generated one
image using this element, and uploaded to fedora people. The image has
passed tests, and has been validated by several people.

So i'm raising that topic to decide what should be the next steps. This
change to generate fedora-atomic images has not already landed into
diskimage-builder. But we have two options here:
- add this element to generic diskimage-builder elements, as i'm doing now
- generate this element internally on magnum. So we can have a directory
in magnum project, called "elements", and have the fedora-atomic element
here. This will give us more control on the element behaviour, and will
allow to update the element without waiting for external reviews.

Once the code for diskimage-builder has landed, another step can be to
periodically generate images using a magnum job, and upload these images
to OpenStack Infra mirrors. Currently the image is based on Fedora F23,
docker-host tree. But different images can be generated if we need a
better option.

As soon as the images are available on internal infra mirrors, the tests
can be changed, to consume these internals images. By this way the tests
can be a bit faster (i know that the bottleneck is on the functional
testing, but if we reduce the download time it can help), and tests can
be more reilable, because we will be removing an exter

Re: [openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Ton Ngo]

Hi Yolanda,
 Thank you for making a huge improvement from the manual process of
building the Fedora Atomic image.
Although Atomic does publish a public OpenStack image that is being
considered in this patch:
https://review.openstack.org/#/c/276232/
in the past we have come to many situations where we need an image with
specific version of certain software
for features or bug fixes (Kubernetes, Docker, Flannel, ...).  So the
automated and customizable build process
will be very helpful.

With respect to where to land the patch, I think diskimage-builder is a
reasonable target.
If it does not land there, Magnum does currently have 2 sets of
diskimage-builder elements for Mesos image
and Ironic image, so it is also reasonable to submit the patch to Magnum.
With the new push to reorganize
into drivers for COE and distro, the elements would be a natural fit for
Fedora Atomic.

   As for periodic image build, it's a good idea to stay current with the
distro, but we should avoid the situation
where something new in the image breaks a COE and we are stuck for awhile
until a fix is made.  So instead of
an automated periodic build, we might want to stage the new image to make
sure it's good before switching.

One question:  I notice the image built by DIB is 871MB, similar to the
manually built image, while the
public image from Atomic is 486MB.  It might be worthwhile to understand
the difference.

Ton Ngo,



From:   Yolanda Robla Mota 
To: 
Date:   03/23/2016 04:12 AM
Subject:[openstack-dev] [magnum] Generate atomic images using
    diskimage-builder



Hi
I wanted to start a discussion on how Fedora Atomic images are being
built. Currently the process for generating the atomic images used on
Magnum is described here:
http://docs.openstack.org/developer/magnum/dev/build-atomic-image.html.
The image needs to be built manually, uploaded to fedorapeople, and then
consumed from there in the magnum tests.
I have been working on a feature to allow diskimage-builder to generate
these images. The code that makes it possible is here:
https://review.openstack.org/287167
This will allow that magnum images are generated on infra, using
diskimage-builder element. This element also has the ability to consume
any tree we need, so images can be customized on demand. I generated one
image using this element, and uploaded to fedora people. The image has
passed tests, and has been validated by several people.

So i'm raising that topic to decide what should be the next steps. This
change to generate fedora-atomic images has not already landed into
diskimage-builder. But we have two options here:
- add this element to generic diskimage-builder elements, as i'm doing now
- generate this element internally on magnum. So we can have a directory
in magnum project, called "elements", and have the fedora-atomic element
here. This will give us more control on the element behaviour, and will
allow to update the element without waiting for external reviews.

Once the code for diskimage-builder has landed, another step can be to
periodically generate images using a magnum job, and upload these images
to OpenStack Infra mirrors. Currently the image is based on Fedora F23,
docker-host tree. But different images can be generated if we need a
better option.

As soon as the images are available on internal infra mirrors, the tests
can be changed, to consume these internals images. By this way the tests
can be a bit faster (i know that the bottleneck is on the functional
testing, but if we reduce the download time it can help), and tests can
be more reilable, because we will be removing an external dependency.

So i'd like to get more feedback on this topic, options and next steps
to achieve the goals. Best

--
Yolanda Robla Mota
Cloud Automation and Distribution Engineer
+34 605641639
yolanda.robla-m...@hpe.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [magnum] Generate atomic images using diskimage-builder

[Yolanda Robla Mota]

Hi
I wanted to start a discussion on how Fedora Atomic images are being 
built. Currently the process for generating the atomic images used on 
Magnum is described here: 
http://docs.openstack.org/developer/magnum/dev/build-atomic-image.html.
The image needs to be built manually, uploaded to fedorapeople, and then 
consumed from there in the magnum tests.
I have been working on a feature to allow diskimage-builder to generate 
these images. The code that makes it possible is here: 
https://review.openstack.org/287167
This will allow that magnum images are generated on infra, using 
diskimage-builder element. This element also has the ability to consume 
any tree we need, so images can be customized on demand. I generated one 
image using this element, and uploaded to fedora people. The image has 
passed tests, and has been validated by several people.


So i'm raising that topic to decide what should be the next steps. This 
change to generate fedora-atomic images has not already landed into 
diskimage-builder. But we have two options here:

- add this element to generic diskimage-builder elements, as i'm doing now
- generate this element internally on magnum. So we can have a directory 
in magnum project, called "elements", and have the fedora-atomic element 
here. This will give us more control on the element behaviour, and will 
allow to update the element without waiting for external reviews.


Once the code for diskimage-builder has landed, another step can be to 
periodically generate images using a magnum job, and upload these images 
to OpenStack Infra mirrors. Currently the image is based on Fedora F23, 
docker-host tree. But different images can be generated if we need a 
better option.


As soon as the images are available on internal infra mirrors, the tests 
can be changed, to consume these internals images. By this way the tests 
can be a bit faster (i know that the bottleneck is on the functional 
testing, but if we reduce the download time it can help), and tests can 
be more reilable, because we will be removing an external dependency.


So i'd like to get more feedback on this topic, options and next steps 
to achieve the goals. Best


--
Yolanda Robla Mota
Cloud Automation and Distribution Engineer
+34 605641639
yolanda.robla-m...@hpe.com


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev