Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-26 Thread Alberto Geniola 
Hi all,

I'm glad to see much interest in this feature. I will prepare and submit
nova specs by the end of this week.
On Oct 23, 2015 5:47 PM, "Markus Zoeller"  wrote:

> Mathieu Gagné  wrote on 10/22/2015 08:08:07 PM:
> > > Could you maybe add a short explanation what the use case is? IOW,
> > > when and why do I (in whatever role) prefetch images?
> > >
> >
> > This previous OpenStack Summit presentation at Paris is a great example
> > of use case:
> > https://www.openstack.org/summit/openstack-paris-summit-2014/session-
> > videos/presentation/cold-start-booting-of-1000-vms-under-10-minutes
> >
> > --
> > Mathieu
>
> Thanks, that's a great presentation with impressive numbers. I wasn't
> aware that there could be the need to distribute new images such often.
>
> Regards, Markus Zoeller (markus_z)
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-23 Thread Markus Zoeller 
Mathieu Gagné  wrote on 10/22/2015 08:08:07 PM:
> > Could you maybe add a short explanation what the use case is? IOW,
> > when and why do I (in whatever role) prefetch images? 
> > 
> 
> This previous OpenStack Summit presentation at Paris is a great example
> of use case:
> https://www.openstack.org/summit/openstack-paris-summit-2014/session-
> videos/presentation/cold-start-booting-of-1000-vms-under-10-minutes
> 
> -- 
> Mathieu

Thanks, that's a great presentation with impressive numbers. I wasn't
aware that there could be the need to distribute new images such often.

Regards, Markus Zoeller (markus_z)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-22 Thread Mathieu Gagné 
On 2015-10-21 10:23 AM, Markus Zoeller wrote:
> Lingxian Kong <anlin.k...@gmail.com> wrote on 10/21/2015 06:44:27 AM:
> 
>> From: Lingxian Kong <anlin.k...@gmail.com>
>> To: "OpenStack Development Mailing List (not for usage questions)" 
>> <openstack-dev@lists.openstack.org>
>> Date: 10/21/2015 06:44 AM
>> Subject: Re: [openstack-dev] [nova-compute][nova][libvirt] Extending 
>> Nova-Compute for image prefetching
>>
>> Hi, Alberto,
>>
>> I'm really interested in your proposal, have you already submitted
>> your spec? or is there any topic for you to have a discussion with
>> Nova team in design summit?
>>
>> I wonder what if the nova compute host use shared storage backend,
>> like NFS or Ceph?
>>
>> I have another suggestion, we can consider adding this capability in
>> nova manage CLI, since it's just may be used by operator or
>> administrator(at least for create/update/delete command), right?
> 
> Could you maybe add a short explanation what the use case is? IOW,
> when and why do I (in whatever role) prefetch images? 
> 

This previous OpenStack Summit presentation at Paris is a great example
of use case:

https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/cold-start-booting-of-1000-vms-under-10-minutes

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-21 Thread Markus Zoeller 
Lingxian Kong <anlin.k...@gmail.com> wrote on 10/21/2015 06:44:27 AM:

> From: Lingxian Kong <anlin.k...@gmail.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: 10/21/2015 06:44 AM
> Subject: Re: [openstack-dev] [nova-compute][nova][libvirt] Extending 
> Nova-Compute for image prefetching
> 
> Hi, Alberto,
> 
> I'm really interested in your proposal, have you already submitted
> your spec? or is there any topic for you to have a discussion with
> Nova team in design summit?
> 
> I wonder what if the nova compute host use shared storage backend,
> like NFS or Ceph?
> 
> I have another suggestion, we can consider adding this capability in
> nova manage CLI, since it's just may be used by operator or
> administrator(at least for create/update/delete command), right?

Could you maybe add a short explanation what the use case is? IOW,
when and why do I (in whatever role) prefetch images? 

Regards, Markus Zoeller (markus_z)



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-20 Thread Lingxian Kong 
Hi, Alberto,

I'm really interested in your proposal, have you already submitted
your spec? or is there any topic for you to have a discussion with
Nova team in design summit?

I wonder what if the nova compute host use shared storage backend,
like NFS or Ceph?

I have another suggestion, we can consider adding this capability in
nova manage CLI, since it's just may be used by operator or
administrator(at least for create/update/delete command), right?

On Thu, Oct 8, 2015 at 7:50 PM, Alberto Geniola
 wrote:
> Hi all,
>
> I'm considering to extend the Nova-Compute API in order to provide
> image-prefetching capabilities to OS.
>
> In order to allow image prefetching, I ended up with the need to add three
> different APIs on the nova-compute nodes:
>
>   1. Trigger an image prefetching
>
>   2. List prefetched images
>
>   3. Delete a prefetched image
>
>
>
> About the point 1 I saw I can re-use the libvirt driver function
> _create_image() to trigger the image prefetching. However, this approach
> will not store any information about the stored image locally. This leads to
> an issue: how do I retrieve a list of already fetched images? A quick and
> simple possibility would be having a local file, storing information about
> the fetched images. Would it be acceptable? Is there any best practice in OS
> community?
>
>
>
> Any ideas?
>
>
> Ty,
>
> Al.
>
>
> --
> Dott. Alberto Geniola
>
>   albertogeni...@gmail.com
>   +39-346-6271105
>   https://www.linkedin.com/in/albertogeniola
>
> Web: http://www.hw4u.it
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards!
---
Lingxian Kong

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-15 Thread John Garbutt 
On 10 October 2015 at 09:35, Alberto Geniola 
wrote:

> Hi Michael,
>
> and thank you for your answer.
>
> Indeed, what I want to do is to add methods to the ImageCache.py module
> (listing, adding, deleting). So far, this module only takes care of image
> deletion: this represents the "cache" of images. Now, I want to populate
> the cache with some images on the hypervisor (as you mention) without
> having any instance running it, yet. The method I want like to add should
> call the appropriate method from the hypervisor driver (let's say libvirt)
> to trigger the image download/creation without starting it (I guess
> something like calling _create_image() should do the trick).
>
> Your question is actually good: how will an user be  able to trigger this
> image caching mechanism?
> My idea is to extend the HTTP Nova Compute API. I would lilke to add a
> resource, let's say "CachedImages", to the API tree. In this way,
> interacting via CRUD operation we should be able to CALL/CAST rpc api and
> interact with the imagecache module.
>
> Is it clearer now? Do you see any problem in this approach?
>

I think prefetching is a problem that needs to be solved.

It would be great if you can submit a nova-spec on this, so we can discuss
it there:
https://github.com/openstack/nova-specs#readme

>From previous discussions, I like the idea of an API to tell a specific
compute node to add an image into its cache. Probably need to specify an
expiry time, for when it gets deleted if never used, etc.

In terms of a CRUD API, I am less sure about the cost/benefit. Thats mostly
because an RPC call in the API always goes badly (with timeouts, etc), and
adding DB tables feels like overkill, and creates a state sync problem. But
maybe I am not seeing the full picture there.

This seems like a good thing to discuss in the spec review, where there
will be more context around the use cases, etc.

Thanks,
johnthetubaguy




>
> On Thu, Oct 8, 2015 at 11:45 PM, Michael Still  wrote:
>
>> I think I'd rephrase your definition of pre-fetched to be honest --
>> something more like "images on this hypervisor node without a currently
>> running instance". So, your operations would become:
>>
>>  - trigger an image prefetching
>>  - list unused base images (and perhaps when they were last used)
>>  - delete an unused image
>>
>> All of that would need to tie into the image cache management code so
>> that its not stomping on your images. In fact, you're probably best of
>> adding all of this as tweaks to the image cache manager anyways.
>>
>> One question though -- who is calling these APIs? Are you adding a
>> central service to orchestrate these calls?
>>
>> Michael
>>
>>
>>
>> On Thu, Oct 8, 2015 at 10:50 PM, Alberto Geniola <
>> albertogeni...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm considering to extend the Nova-Compute API in order to provide
>>> image-prefetching capabilities to OS.
>>>
>>> In order to allow image prefetching, I ended up with the need to add
>>> three different APIs on the nova-compute nodes:
>>>
>>>   1. Trigger an image prefetching
>>>
>>>   2. List prefetched images
>>>
>>>   3. Delete a prefetched image
>>>
>>>
>>>
>>> About the point 1 I saw I can re-use the libvirt driver function
>>> _create_image() to trigger the image prefetching. However, this approach
>>> will not store any information about the stored image locally. This leads
>>> to an issue: how do I retrieve a list of already fetched images? A quick
>>> and simple possibility would be having a local file, storing information
>>> about the fetched images. Would it be acceptable? Is there any best
>>> practice in OS community?
>>>
>>>
>>>
>>> Any ideas?
>>>
>>>
>>> Ty,
>>>
>>> Al.
>>>
>>> --
>>> Dott. Alberto Geniola
>>>
>>>   albertogeni...@gmail.com
>>>   +39-346-6271105
>>>   https://www.linkedin.com/in/albertogeniola
>>>
>>> Web: http://www.hw4u.it
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Rackspace Australia
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Dott. Alberto Geniola
>
>   albertogeni...@gmail.com
>   +39-346-6271105
>   https://www.linkedin.com/in/albertogeniola
>
> Web: http://www.hw4u.it
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-10 Thread Alberto Geniola 
Hi Michael,

and thank you for your answer.

Indeed, what I want to do is to add methods to the ImageCache.py module
(listing, adding, deleting). So far, this module only takes care of image
deletion: this represents the "cache" of images. Now, I want to populate
the cache with some images on the hypervisor (as you mention) without
having any instance running it, yet. The method I want like to add should
call the appropriate method from the hypervisor driver (let's say libvirt)
to trigger the image download/creation without starting it (I guess
something like calling _create_image() should do the trick).

Your question is actually good: how will an user be  able to trigger this
image caching mechanism?
My idea is to extend the HTTP Nova Compute API. I would lilke to add a
resource, let's say "CachedImages", to the API tree. In this way,
interacting via CRUD operation we should be able to CALL/CAST rpc api and
interact with the imagecache module.

Is it clearer now? Do you see any problem in this approach?

Regards,

Alberto.




On Thu, Oct 8, 2015 at 11:45 PM, Michael Still  wrote:

> I think I'd rephrase your definition of pre-fetched to be honest --
> something more like "images on this hypervisor node without a currently
> running instance". So, your operations would become:
>
>  - trigger an image prefetching
>  - list unused base images (and perhaps when they were last used)
>  - delete an unused image
>
> All of that would need to tie into the image cache management code so that
> its not stomping on your images. In fact, you're probably best of adding
> all of this as tweaks to the image cache manager anyways.
>
> One question though -- who is calling these APIs? Are you adding a central
> service to orchestrate these calls?
>
> Michael
>
>
>
> On Thu, Oct 8, 2015 at 10:50 PM, Alberto Geniola  > wrote:
>
>> Hi all,
>>
>> I'm considering to extend the Nova-Compute API in order to provide
>> image-prefetching capabilities to OS.
>>
>> In order to allow image prefetching, I ended up with the need to add
>> three different APIs on the nova-compute nodes:
>>
>>   1. Trigger an image prefetching
>>
>>   2. List prefetched images
>>
>>   3. Delete a prefetched image
>>
>>
>>
>> About the point 1 I saw I can re-use the libvirt driver function
>> _create_image() to trigger the image prefetching. However, this approach
>> will not store any information about the stored image locally. This leads
>> to an issue: how do I retrieve a list of already fetched images? A quick
>> and simple possibility would be having a local file, storing information
>> about the fetched images. Would it be acceptable? Is there any best
>> practice in OS community?
>>
>>
>>
>> Any ideas?
>>
>>
>> Ty,
>>
>> Al.
>>
>> --
>> Dott. Alberto Geniola
>>
>>   albertogeni...@gmail.com
>>   +39-346-6271105
>>   https://www.linkedin.com/in/albertogeniola
>>
>> Web: http://www.hw4u.it
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Rackspace Australia
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Dott. Alberto Geniola

  albertogeni...@gmail.com
  +39-346-6271105
  https://www.linkedin.com/in/albertogeniola

Web: http://www.hw4u.it
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-08 Thread Alberto Geniola 
Hi all,

I'm considering to extend the Nova-Compute API in order to provide
image-prefetching capabilities to OS.

In order to allow image prefetching, I ended up with the need to add three
different APIs on the nova-compute nodes:

  1. Trigger an image prefetching

  2. List prefetched images

  3. Delete a prefetched image



About the point 1 I saw I can re-use the libvirt driver function
_create_image() to trigger the image prefetching. However, this approach
will not store any information about the stored image locally. This leads
to an issue: how do I retrieve a list of already fetched images? A quick
and simple possibility would be having a local file, storing information
about the fetched images. Would it be acceptable? Is there any best
practice in OS community?



Any ideas?


Ty,

Al.

-- 
Dott. Alberto Geniola

  albertogeni...@gmail.com
  +39-346-6271105
  https://www.linkedin.com/in/albertogeniola

Web: http://www.hw4u.it
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

2015-10-08 Thread Michael Still 
I think I'd rephrase your definition of pre-fetched to be honest --
something more like "images on this hypervisor node without a currently
running instance". So, your operations would become:

 - trigger an image prefetching
 - list unused base images (and perhaps when they were last used)
 - delete an unused image

All of that would need to tie into the image cache management code so that
its not stomping on your images. In fact, you're probably best of adding
all of this as tweaks to the image cache manager anyways.

One question though -- who is calling these APIs? Are you adding a central
service to orchestrate these calls?

Michael



On Thu, Oct 8, 2015 at 10:50 PM, Alberto Geniola 
wrote:

> Hi all,
>
> I'm considering to extend the Nova-Compute API in order to provide
> image-prefetching capabilities to OS.
>
> In order to allow image prefetching, I ended up with the need to add three
> different APIs on the nova-compute nodes:
>
>   1. Trigger an image prefetching
>
>   2. List prefetched images
>
>   3. Delete a prefetched image
>
>
>
> About the point 1 I saw I can re-use the libvirt driver function
> _create_image() to trigger the image prefetching. However, this approach
> will not store any information about the stored image locally. This leads
> to an issue: how do I retrieve a list of already fetched images? A quick
> and simple possibility would be having a local file, storing information
> about the fetched images. Would it be acceptable? Is there any best
> practice in OS community?
>
>
>
> Any ideas?
>
>
> Ty,
>
> Al.
>
> --
> Dott. Alberto Geniola
>
>   albertogeni...@gmail.com
>   +39-346-6271105
>   https://www.linkedin.com/in/albertogeniola
>
> Web: http://www.hw4u.it
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Rackspace Australia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev