Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Waines, Greg]

In-lined below,
Greg

From: "Csatari, Gergely (Nokia - HU/Budapest)" 
Date: Monday, July 2, 2018 at 7:15 AM
To: Greg Waines , 
"openstack-dev@lists.openstack.org" , 
"edge-comput...@lists.openstack.org" 
Subject: RE: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

Going inline.

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Friday, June 29, 2018 4:25 AM


In-lined comments / questions below,
Greg.

From: "Csatari, Gergely (Nokia - HU/Budapest)" 
mailto:gergely.csat...@nokia.com>>
Date: Thursday, June 28, 2018 at 3:35 AM



Hi,

I’ve added the following pros and cons to the different options:




  *   One Glance with multiple backends 
[1]
[Greg]
I’m not sure I understand this option.

Is each Glance Backend completely independent ?   e.g. when I do a “glance 
image-create ...” am I specifying a backend and that’s where the image is to be 
stored ?
This is what I was originally thinking.
So I was thinking that synchronization of images to Edge Clouds is simply done 
by doing “glance image-create ...” to the appropriate backends.

But then you say “The syncronisation of the image data is the responsibility of 
the backend (eg.: CEPH).” ... which makes it sound like my thinking above is 
wrong and the Backends are NOT completely independent, but instead in some sort 
of replication configuration ... is this leveraging ceph replication factor or 
something (for example) ?
[G0]: According to my understanding the backends are in a replication 
configuration in this case. Jokke, am I right?

 *   Pros:
*   Relatively easy to implement based on the current Glance 
architecture
 *   Cons:
*   Requires the same Glance backend in every edge cloud instance
*   Requires the same OpenStack version in every edge cloud instance 
(apart from during upgrade)
*   Sensitivity for network connection loss is not clear
[Greg] I could be wrong, but even though the OpenStack services in the edge 
clouds are using the images in their glance backend with a direct URL,
I think the OpenStack services (e.g. nova) still need to get the direct URL via 
the Glance API which is ONLY available at the central site.
So don’t think this option supports autonomy of edge Subcloud when connectivity 
is lost to central site.
[G0]: Can’t the url point to the local Glance backend somehow?
[Greg] Need some input from Nova or Glance guy,
 but believe that Nova must still use Glance API to get access to 
the image, however if the storage is remote, there can be some backend 
optimization between glance and nova, e.g. a direct URL to the image.

  *   Several Glances with an independent syncronisation service, sych via 
Glance API 
[2]
 *   Pros:
*   Every edge cloud instance can have a different Glance backend
*   Can support multiple OpenStack versions in the different edge cloud 
instances
*   Can be extended to support multiple VIM types
 *   Cons:
*   Needs a new synchronisation service
[Greg] Don’t believe this is a big con ... suspect we are going to need this 
new synchronization service for synchronizing resources of a number of other 
openstack services ... not just glance.
[G0]: I agree, it is not a big con, but it is a con  Should I add some note 
saying, that a synch service is most probably needed anyway?

  *   Several Glances with an independent syncronisation service, synch using 
the backend 
[3]
[Greg] This option seems a little odd to me.
We are synching the GLANCE DB via some new synchronization service, but 
synching the Images themselves via the backend ... I think that would be tricky 
to ensure consistency.
[G0]: Yes, there is a place for errors here.

 *   Pros:
*   I could not find any
 *   Cons:
*   Needs a new synchronisation service



  *   One Glance and multiple Glance API servers 
[4]
 *   Pros:
*   Implicitly location aware
 *   Cons:
*   First usage of an image always takes a long time
*   In case of network connection error to the central Galnce Nova will 
have access to the images, but will not be able to figure out if the user have 
rights to use the image and will not have path to the images data
[Greg] Yeah we tripped over the issue that although the Glance API can cache 
the image itself, it does NOT cache the image meta data (which I am guessing 
has info like “user access” etc.) 

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Csatari, Gergely (Nokia - HU/Budapest)]

Hi,

Going inline.

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Friday, June 29, 2018 4:25 AM

In-lined comments / questions below,
Greg.

From: "Csatari, Gergely (Nokia - HU/Budapest)" 
mailto:gergely.csat...@nokia.com>>
Date: Thursday, June 28, 2018 at 3:35 AM


Hi,

I’ve added the following pros and cons to the different options:



  *   One Glance with multiple backends 
[1]
[Greg]
I’m not sure I understand this option.

Is each Glance Backend completely independent ?   e.g. when I do a “glance 
image-create ...” am I specifying a backend and that’s where the image is to be 
stored ?
This is what I was originally thinking.
So I was thinking that synchronization of images to Edge Clouds is simply done 
by doing “glance image-create ...” to the appropriate backends.

But then you say “The syncronisation of the image data is the responsibility of 
the backend (eg.: CEPH).” ... which makes it sound like my thinking above is 
wrong and the Backends are NOT completely independent, but instead in some sort 
of replication configuration ... is this leveraging ceph replication factor or 
something (for example) ?
[G0]: According to my understanding the backends are in a replication 
configuration in this case. Jokke, am I right?

 *   Pros:
*   Relatively easy to implement based on the current Glance 
architecture
 *   Cons:
*   Requires the same Glance backend in every edge cloud instance
*   Requires the same OpenStack version in every edge cloud instance 
(apart from during upgrade)
*   Sensitivity for network connection loss is not clear
[Greg] I could be wrong, but even though the OpenStack services in the edge 
clouds are using the images in their glance backend with a direct URL,
I think the OpenStack services (e.g. nova) still need to get the direct URL via 
the Glance API which is ONLY available at the central site.
So don’t think this option supports autonomy of edge Subcloud when connectivity 
is lost to central site.
[G0]: Can’t the url point to the local Glance backend somehow?

  *   Several Glances with an independent syncronisation service, sych via 
Glance API 
[2]
 *   Pros:
*   Every edge cloud instance can have a different Glance backend
*   Can support multiple OpenStack versions in the different edge cloud 
instances
*   Can be extended to support multiple VIM types
 *   Cons:
*   Needs a new synchronisation service
[Greg] Don’t believe this is a big con ... suspect we are going to need this 
new synchronization service for synchronizing resources of a number of other 
openstack services ... not just glance.
[G0]: I agree, it is not a big con, but it is a con  Should I add some note 
saying, that a synch service is most probably needed anyway?

  *   Several Glances with an independent syncronisation service, synch using 
the backend 
[3]
[Greg] This option seems a little odd to me.
We are synching the GLANCE DB via some new synchronization service, but 
synching the Images themselves via the backend ... I think that would be tricky 
to ensure consistency.
[G0]: Yes, there is a place for errors here.

 *   Pros:
*   I could not find any
 *   Cons:
*   Needs a new synchronisation service


  *   One Glance and multiple Glance API servers 
[4]
 *   Pros:
*   Implicitly location aware
 *   Cons:
*   First usage of an image always takes a long time
*   In case of network connection error to the central Galnce Nova will 
have access to the images, but will not be able to figure out if the user have 
rights to use the image and will not have path to the images data
[Greg] Yeah we tripped over the issue that although the Glance API can cache 
the image itself, it does NOT cache the image meta data (which I am guessing 
has info like “user access” etc.) ... so this option improves latency of access 
to image itself but does NOT provide autonomy.

We plan on looking at options to resolve this, as we like the “implicit 
location awareness” of this option ... and believe it is an option that some 
customers will like.
If anyone has any ideas ?
Are these correct? Do I miss anything?

Thanks,
Gerg0

[1]: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#One_Glance_with_multiple_backends
[2]: 

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Waines, Greg]

In-lined comments / questions below,
Greg.



From: "Csatari, Gergely (Nokia - HU/Budapest)" 
mailto:gergely.csat...@nokia.com>>
Date: Thursday, June 28, 2018 at 3:35 AM
To: "ekuv...@redhat.com" 
mailto:ekuv...@redhat.com>>, Greg Waines 
mailto:greg.wai...@windriver.com>>, 
"openstack-dev@lists.openstack.org" 
mailto:openstack-dev@lists.openstack.org>>, 
"edge-comput...@lists.openstack.org" 
mailto:edge-comput...@lists.openstack.org>>
Subject: RE: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

I’ve added the following pros and cons to the different options:




  *   One Glance with multiple backends 
[1]
[Greg]
I’m not sure I understand this option.

Is each Glance Backend completely independent ?   e.g. when I do a “glance 
image-create ...” am I specifying a backend and that’s where the image is to be 
stored ?
This is what I was originally thinking.
So I was thinking that synchronization of images to Edge Clouds is simply done 
by doing “glance image-create ...” to the appropriate backends.

But then you say “The syncronisation of the image data is the responsibility of 
the backend (eg.: CEPH).” ... which makes it sound like my thinking above is 
wrong and the Backends are NOT completely independent, but instead in some sort 
of replication configuration ... is this leveraging ceph replication factor or 
something (for example) ?



 *   Pros:
*   Relatively easy to implement based on the current Glance 
architecture
 *   Cons:
*   Requires the same Glance backend in every edge cloud instance
*   Requires the same OpenStack version in every edge cloud instance 
(apart from during upgrade)
*   Sensitivity for network connection loss is not clear
[Greg] I could be wrong, but even though the OpenStack services in the edge 
clouds are using the images in their glance backend with a direct URL,
I think the OpenStack services (e.g. nova) still need to get the direct URL via 
the Glance API which is ONLY available at the central site.
So don’t think this option supports autonomy of edge Subcloud when connectivity 
is lost to central site.



  *   Several Glances with an independent syncronisation service, sych via 
Glance API 
[2]
 *   Pros:
*   Every edge cloud instance can have a different Glance backend
*   Can support multiple OpenStack versions in the different edge cloud 
instances
*   Can be extended to support multiple VIM types
 *   Cons:
*   Needs a new synchronisation service
[Greg] Don’t believe this is a big con ... suspect we are going to need this 
new synchronization service for synchronizing resources of a number of other 
openstack services ... not just glance.



  *   Several Glances with an independent syncronisation service, synch using 
the backend 
[3]
[Greg] This option seems a little odd to me.
We are synching the GLANCE DB via some new synchronization service, but 
synching the Images themselves via the backend ... I think that would be tricky 
to ensure consistency.
 *   Pros:
*   I could not find any
 *   Cons:
*   Needs a new synchronisation service



  *   One Glance and multiple Glance API servers 
[4]
 *   Pros:
*   Implicitly location aware
 *   Cons:
*   First usage of an image always takes a long time
*   In case of network connection error to the central Galnce Nova will 
have access to the images, but will not be able to figure out if the user have 
rights to use the image and will not have path to the images data
[Greg] Yeah we tripped over the issue that although the Glance API can cache 
the image itself, it does NOT cache the image meta data (which I am guessing 
has info like “user access” etc.) ... so this option improves latency of access 
to image itself but does NOT provide autonomy.

We plan on looking at options to resolve this, as we like the “implicit 
location awareness” of this option ... and believe it is an option that some 
customers will like.
If anyone has any ideas ?
Are these correct? Do I miss anything?

Thanks,
Gerg0

[1]: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#One_Glance_with_multiple_backends
[2]: 

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Csatari, Gergely (Nokia - HU/Budapest)]

Hi,

I’ve added the following pros and cons to the different options:

  *   One Glance with multiple backends 
[1]
 *   Pros:
*   Relatively easy to implement based on the current Glance 
architecture
 *   Cons:
*   Requires the same Glance backend in every edge cloud instance
*   Requires the same OpenStack version in every edge cloud instance 
(apart from during upgrade)
*   Sensitivity for network connection loss is not clear
  *   Several Glances with an independent syncronisation service, sych via 
Glance API 
[2]
 *   Pros:
*   Every edge cloud instance can have a different Glance backend
*   Can support multiple OpenStack versions in the different edge cloud 
instances
*   Can be extended to support multiple VIM types
 *   Cons:
*   Needs a new synchronisation service
  *   Several Glances with an independent syncronisation service, synch using 
the backend 
[3]
 *   Pros:
*   I could not find any
 *   Cons:
*   Needs a new synchronisation service
  *   One Glance and multiple Glance API servers 
[4]
 *   Pros:
*   Implicitly location aware
 *   Cons:
*   First usage of an image always takes a long time
*   In case of network connection error to the central Galnce Nova will 
have access to the images, but will not be able to figure out if the user have 
rights to use the image and will not have path to the images data
Are these correct? Do I miss anything?

Thanks,
Gerg0

[1]: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#One_Glance_with_multiple_backends
[2]: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#Several_Glances_with_an_independent_syncronisation_service.2C_sych_via_Glance_API
[3]: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#Several_Glances_with_an_independent_syncronisation_service.2C_synch_using_the_backend
[4]: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#One_Glance_and_multiple_Glance_API_servers






From: Csatari, Gergely (Nokia - HU/Budapest)
Sent: Monday, June 11, 2018 4:29 PM
To: Waines, Greg ; OpenStack Development Mailing 
List (not for usage questions) ; 
edge-comput...@lists.openstack.org
Subject: RE: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

Thanks for the comments.
I’ve updated the wiki: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#Several_Glances_with_an_independent_syncronisation_service.2C_synch_using_the_backend

Br,
Gerg0

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Friday, June 8, 2018 1:46 PM
To: Csatari, Gergely (Nokia - HU/Budapest) 
mailto:gergely.csat...@nokia.com>>; OpenStack 
Development Mailing List (not for usage questions) 
mailto:openstack-dev@lists.openstack.org>>; 
edge-comput...@lists.openstack.org
Subject: Re: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Responses in-lined below,
Greg.

From: "Csatari, Gergely (Nokia - HU/Budapest)" 
mailto:gergely.csat...@nokia.com>>
Date: Friday, June 8, 2018 at 3:39 AM
To: Greg Waines mailto:greg.wai...@windriver.com>>, 
"openstack-dev@lists.openstack.org" 
mailto:openstack-dev@lists.openstack.org>>, 
"edge-comput...@lists.openstack.org" 
mailto:edge-comput...@lists.openstack.org>>
Subject: RE: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

Going inline.

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Thursday, June 7, 2018 2:24 PM
I had some additional questions/comments on the Image Synchronization Options ( 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment ):


One Glance with multiple backends

  *   In this scenario, are all Edge Clouds simply configured with the one 
central glance for its GLANCE ENDPOINT ?
 *   i.e. GLANCE is a typical shared service in a multi-region environment ?

[G0]: In my understanding yes.


  *   If so,
how does this OPTION support the requirement for Edge Cloud Operation when 
disconnected from Central Location ?

[G0]: This is an open question for me also.


Several Glances with an independent synchronization service(PUSH)

  *   I refer to this as the PUSH 

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Kristi Nikolla]

Answering here questions related to mixmatch, as I'm the main developer.

- If I understood [[1](https://mixmatch.readthedocs.io/en/latest/)] correctly 
mixmatch can help Nova to attach a remote volume, but it will not help in 
synchronizing the images. is this true?

Correct, it will not synchronize images. What it does is forward API requests 
across OpenStack clouds, allowing Nova to fetch a remote image. It could be 
enhanced to "cache" the image, in which case it would be the PULL model. In 
this architecture, you would have mixmatch acting as a proxy to one or multiple 
Glance API servers (some remote), and you could even potentially forego having 
a Glance at all in the edge cloud.

- Not sure ... but I didn’t think this was the model being used in mixmatch ... 
thought mixmatch was more the PULL model (below)

- [G0]: Yes, this is more or less my understanding. I remove the mixmatch 
reference from this chapter.

Rather than remove it completely you could move it to the correct section. :)

‐‐‐ Original Message ‐‐‐
On June 11, 2018 10:28 AM, Csatari, Gergely (Nokia - HU/Budapest) 
 wrote:

> Hi,
>
> Thanks for the comments.
>
> I’ve updated the wiki: 
> https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#Several_Glances_with_an_independent_syncronisation_service.2C_synch_using_the_backend
>
> Br,
>
> Gerg0
>
> [ ]
>
> From: Waines, Greg [mailto:greg.wai...@windriver.com]
> Sent: Friday, June 8, 2018 1:46 PM
> To: Csatari, Gergely (Nokia - HU/Budapest) ; 
> OpenStack Development Mailing List (not for usage questions) 
> ; edge-comput...@lists.openstack.org
> Subject: Re: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
> architectures for image synchronisation
>
> Responses in-lined below,
>
> Greg.
>
> From: "Csatari, Gergely (Nokia - HU/Budapest)" 
> Date: Friday, June 8, 2018 at 3:39 AM
> To: Greg Waines , 
> "openstack-dev@lists.openstack.org" , 
> "edge-comput...@lists.openstack.org" 
> Subject: RE: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
> architectures for image synchronisation
>
> Hi,
>
> Going inline.
>
> From: Waines, Greg [mailto:greg.wai...@windriver.com]
> Sent: Thursday, June 7, 2018 2:24 PM
>
> I had some additional questions/comments on the Image Synchronization Options 
> ( https://wiki.openstack.org/wiki/Image_handling_in_edge_environment ):
>
> One Glance with multiple backends
>
> - In this scenario, are all Edge Clouds simply configured with the one 
> central glance for its GLANCE ENDPOINT ?
>
> - i.e. GLANCE is a typical shared service in a multi-region environment ?
>
> [G0]: In my understanding yes.
>
> - If so,
> how does this OPTION support the requirement for Edge Cloud Operation when 
> disconnected from Central Location ?
>
> [G0]: This is an open question for me also.
>
> Several Glances with an independent synchronization service(PUSH)
>
> - I refer to this as the PUSH model
>
> - I don’t believe you have to ( or necessarily should) rely on the backend to 
> do the synchronization of the images
>
> - i.e. the ‘Synch Service’ could do this strictly through Glance REST APIs
> (making it independent of the particular Glance backend ... and allowing the 
> Glance Backends at Central and Edge sites to actually be different)
>
> [G0]: Okay, I can update the wiki to reflect this. Should we keep the 
> “synchronization by the backend” option as an other alternative?
> [Greg] Yeah we should keep it as an alternative.
>
> - I think the ‘Synch Service’ MUST be able to support ‘selective/multicast’ 
> distribution of Images from Central to Edge for Image Synchronization
>
> - i.e. you don’t want Central Site pushing ALL images to ALL Edge Sites ... 
> especially for the small Edge Sites
>
> [G0]: Yes, the question is how to define these synchronization policies.
> [Greg] Agreed ... we’ve had some very high-level discussions with end users, 
> but haven’t put together a proposal yet.
>
> - Not sure ... but I didn’t think this was the model being used in mixmatch 
> ... thought mixmatch was more the PULL model (below)
>
> [G0]: Yes, this is more or less my understanding. I remove the mixmatch 
> reference from this chapter.
>
> One Glance and multiple Glance API Servers   (PULL)
>
> - I refer to this as the PULL model
>
> - This is the current model supported in StarlingX’s Distributed Cloud 
> sub-project
>
> - We run glance-api on all Edge Clouds ... that talk to glance-registry on 
> the Central Cloud, and
>
> - We have glance-api setup for caching such that only the first access to an 
> particular image incurs the latency of the image transfer from Central to Edge
>
> [G0]: Do you do image caching in Glance API or do you rely in the image cache 
> in Nova? In the Forum session there were some discussions about this and I 
> think the conclusion was that using the image cache of Nova is enough.
> [Greg] We enabled image caching in the Glance API.
>  I believe that Nova Image Caching caches at the 

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Csatari, Gergely (Nokia - HU/Budapest)]

Hi,

Thanks for the comments.
I’ve updated the wiki: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment#Several_Glances_with_an_independent_syncronisation_service.2C_synch_using_the_backend

Br,
Gerg0

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Friday, June 8, 2018 1:46 PM
To: Csatari, Gergely (Nokia - HU/Budapest) ; 
OpenStack Development Mailing List (not for usage questions) 
; edge-comput...@lists.openstack.org
Subject: Re: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Responses in-lined below,
Greg.

From: "Csatari, Gergely (Nokia - HU/Budapest)" 
mailto:gergely.csat...@nokia.com>>
Date: Friday, June 8, 2018 at 3:39 AM
To: Greg Waines mailto:greg.wai...@windriver.com>>, 
"openstack-dev@lists.openstack.org" 
mailto:openstack-dev@lists.openstack.org>>, 
"edge-comput...@lists.openstack.org" 
mailto:edge-comput...@lists.openstack.org>>
Subject: RE: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

Going inline.

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Thursday, June 7, 2018 2:24 PM

I had some additional questions/comments on the Image Synchronization Options ( 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment ):


One Glance with multiple backends

  *   In this scenario, are all Edge Clouds simply configured with the one 
central glance for its GLANCE ENDPOINT ?
 *   i.e. GLANCE is a typical shared service in a multi-region environment ?

[G0]: In my understanding yes.


  *   If so,
how does this OPTION support the requirement for Edge Cloud Operation when 
disconnected from Central Location ?

[G0]: This is an open question for me also.


Several Glances with an independent synchronization service(PUSH)

  *   I refer to this as the PUSH model
  *   I don’t believe you have to ( or necessarily should) rely on the backend 
to do the synchronization of the images
 *   i.e. the ‘Synch Service’ could do this strictly through Glance REST 
APIs
(making it independent of the particular Glance backend ... and allowing the 
Glance Backends at Central and Edge sites to actually be different)
[G0]: Okay, I can update the wiki to reflect this. Should we keep the 
“synchronization by the backend” option as an other alternative?
[Greg] Yeah we should keep it as an alternative.

  *   I think the ‘Synch Service’ MUST be able to support ‘selective/multicast’ 
distribution of Images from Central to Edge for Image Synchronization
 *   i.e. you don’t want Central Site pushing ALL images to ALL Edge Sites 
... especially for the small Edge Sites
[G0]: Yes, the question is how to define these synchronization policies.
[Greg] Agreed ... we’ve had some very high-level discussions with end users, 
but haven’t put together a proposal yet.

  *   Not sure ... but I didn’t think this was the model being used in mixmatch 
... thought mixmatch was more the PULL model (below)

[G0]: Yes, this is more or less my understanding. I remove the mixmatch 
reference from this chapter.

One Glance and multiple Glance API Servers   (PULL)

  *   I refer to this as the PULL model
  *   This is the current model supported in StarlingX’s Distributed Cloud 
sub-project
 *   We run glance-api on all Edge Clouds ... that talk to glance-registry 
on the Central Cloud, and
 *   We have glance-api setup for caching such that only the first access 
to an particular image incurs the latency of the image transfer from Central to 
Edge
[G0]: Do you do image caching in Glance API or do you rely in the image cache 
in Nova? In the Forum session there were some discussions about this and I 
think the conclusion was that using the image cache of Nova is enough.
[Greg] We enabled image caching in the Glance API.
 I believe that Nova Image Caching caches at the compute node ... 
this would work ok for all-in-one edge clouds or small edge clouds.
 But glance-api caching caches at the edge cloud level, so works 
better for large edge clouds with lots of compute nodes.

  *
  *   this PULL model affectively implements the location aware synchronization 
you talk about below,  (i.e. synchronise images only to those cloud instances 
where they are needed)?


In StarlingX Distributed Cloud,
We plan on supporting both the PUSH and PULL model ... suspect there are use 
cases for both.

[G0]: This means that you need an architecture supporting both.
Just for my curiosity what is the use case for the pull model once you have the 
push model in place?
[Greg] The PULL model certainly results in the most efficient distribution of 
images ... basically images are distributed ONLY to edge clouds that explicitly 
use the image.
Also if the use case is NOT concerned about incurring the latency of the image 
transfer from Central to Edge on the FIRST use 

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Csatari, Gergely (Nokia - HU/Budapest)]

Hi, 

Going inline. 

-Original Message-
From: Erno Kuvaja [mailto:ekuv...@redhat.com] 
Sent: Friday, June 8, 2018 3:18 PM

Hi,

Answering inline.

Best,
Erno "jokke" Kuvaja

On Thu, Jun 7, 2018 at 11:49 AM, Csatari, Gergely (Nokia -
HU/Budapest)  wrote:
> Hi,
>
>
>
> I did some work ont he figures and realised, that I have some 
> questions related to the alternative options:
>
>
>
> Multiple backends option:
>
> What is the API between Glance and the Glance backends?
glance_store library
> How is it possible to implement location aware synchronisation 
> (synchronise images only to those cloud instances where they are needed)?
This needs bit of hooking. We need to update the locations into Glance once the 
replication has happened.
[G0]: Okay, but how to avoid the replication to sites where the image is not 
needed?

> Is it possible to have different OpenStack versions in the different 
> cloud instances?
In my understanding it's not supported to mix versions within OpenStack cloud 
apart from during upgrade.
[G0]: Understood. This might be a problem ont he long run. With lots of edge 
cloud instance it can not be guaranteed, that all of them are upgraded in one 
go. 

> Can a cloud instance use the locally synchronised images in case of a 
> network connection break?
That depends a lot of the implementation. If there is local glance node with 
replicated db and store, yes.
[G0]: So we need a replicated Glance DB, a store and a backend in every edge 
cloud instance for this?
How the database would be syncronised in this case?

> Is it possible to implement this without storing database credentials 
> ont he edge cloud instances?
Again depending of the deployment. You definitely cannot have both, access 
during network outage and access without db credentials. if one needs to have 
local access of images without db credentials, there is always possibility for 
the local Ceph back-end with remote glance-api node. In this case Nova can talk 
directly to the local Ceph back-end and communicate with centralized glance-api 
that has the credentials to the db. The problem with loosing the network in 
this scenario is that Nova will have no idea if the user has rights to use the 
image or not and it will not know the path to that image's data.
[G0]: Okay

> Independent synchronisation service:
>
> If I understood [1] correctly mixmatch can help Nova to attach a 
> remote volume, but it will not help in synchronizing the images. is this true?
>
> As I promised in the Edge Compute Group call I plan to organize an IRC 
> review meeting to check the wiki. Please indicate your availability in [2].
>
> [1]: https://mixmatch.readthedocs.io/en/latest/
>
> [2]: https://doodle.com/poll/bddg65vyh4qwxpk5
[G0]: Please add your availability here.

Thanks, 
Gerg0 

>
>
>
> Br,
>
> Gerg0
>
>
>
> From: Csatari, Gergely (Nokia - HU/Budapest)
> Sent: Wednesday, May 23, 2018 8:59 PM
> To: OpenStack Development Mailing List (not for usage questions) 
> ; 
> edge-comput...@lists.openstack.org
> Subject: [edge][glance]: Wiki of the possible architectures for image 
> synchronisation
>
>
>
> Hi,
>
>
>
> Here I send the wiki page [1] where I summarize what I understood from 
> the Forum session about image synchronisation in edge environment [2], [3].
>
>
>
> Please check and correct/comment.
>
>
>
> Thanks,
>
> Gerg0
>
>
>
>
>
> [1]: 
> https://wiki.openstack.org/wiki/Image_handling_in_edge_environment
>
> [2]: https://etherpad.openstack.org/p/yvr-edge-cloud-images
>
> [3]:
> https://www.openstack.org/summit/vancouver-2018/summit-schedule/events
> /21768/image-handling-in-an-edge-cloud-infrastructure
>
>
> ___
> Edge-computing mailing list
> edge-comput...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Erno Kuvaja]

Hi,

Answering inline.

Best,
Erno "jokke" Kuvaja

On Thu, Jun 7, 2018 at 11:49 AM, Csatari, Gergely (Nokia -
HU/Budapest)  wrote:
> Hi,
>
>
>
> I did some work ont he figures and realised, that I have some questions
> related to the alternative options:
>
>
>
> Multiple backends option:
>
> What is the API between Glance and the Glance backends?
glance_store library
> How is it possible to implement location aware synchronisation (synchronise
> images only to those cloud instances where they are needed)?
This needs bit of hooking. We need to update the locations into Glance
once the replication has happened.
> Is it possible to have different OpenStack versions in the different cloud
> instances?
In my understanding it's not supported to mix versions within
OpenStack cloud apart from during upgrade.
> Can a cloud instance use the locally synchronised images in case of a
> network connection break?
That depends a lot of the implementation. If there is local glance
node with replicated db and store, yes.
> Is it possible to implement this without storing database credentials ont he
> edge cloud instances?
Again depending of the deployment. You definitely cannot have both,
access during network outage and access without db credentials. if one
needs to have local access of images without db credentials, there is
always possibility for the local Ceph back-end with remote glance-api
node. In this case Nova can talk directly to the local Ceph back-end
and communicate with centralized glance-api that has the credentials
to the db. The problem with loosing the network in this scenario is
that Nova will have no idea if the user has rights to use the image or
not and it will not know the path to that image's data.
>
>
>
> Independent synchronisation service:
>
> If I understood [1] correctly mixmatch can help Nova to attach a remote
> volume, but it will not help in synchronizing the images. is this true?
>
>
>
>
>
> As I promised in the Edge Compute Group call I plan to organize an IRC
> review meeting to check the wiki. Please indicate your availability in [2].
>
>
>
> [1]: https://mixmatch.readthedocs.io/en/latest/
>
> [2]: https://doodle.com/poll/bddg65vyh4qwxpk5
>
>
>
> Br,
>
> Gerg0
>
>
>
> From: Csatari, Gergely (Nokia - HU/Budapest)
> Sent: Wednesday, May 23, 2018 8:59 PM
> To: OpenStack Development Mailing List (not for usage questions)
> ; edge-comput...@lists.openstack.org
> Subject: [edge][glance]: Wiki of the possible architectures for image
> synchronisation
>
>
>
> Hi,
>
>
>
> Here I send the wiki page [1] where I summarize what I understood from the
> Forum session about image synchronisation in edge environment [2], [3].
>
>
>
> Please check and correct/comment.
>
>
>
> Thanks,
>
> Gerg0
>
>
>
>
>
> [1]: https://wiki.openstack.org/wiki/Image_handling_in_edge_environment
>
> [2]: https://etherpad.openstack.org/p/yvr-edge-cloud-images
>
> [3]:
> https://www.openstack.org/summit/vancouver-2018/summit-schedule/events/21768/image-handling-in-an-edge-cloud-infrastructure
>
>
> ___
> Edge-computing mailing list
> edge-comput...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Waines, Greg]

Responses in-lined below,
Greg.

From: "Csatari, Gergely (Nokia - HU/Budapest)" 
Date: Friday, June 8, 2018 at 3:39 AM
To: Greg Waines , 
"openstack-dev@lists.openstack.org" , 
"edge-comput...@lists.openstack.org" 
Subject: RE: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

Going inline.

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Thursday, June 7, 2018 2:24 PM


I had some additional questions/comments on the Image Synchronization Options ( 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment ):


One Glance with multiple backends

  *   In this scenario, are all Edge Clouds simply configured with the one 
central glance for its GLANCE ENDPOINT ?
 *   i.e. GLANCE is a typical shared service in a multi-region environment ?

[G0]: In my understanding yes.


  *   If so,
how does this OPTION support the requirement for Edge Cloud Operation when 
disconnected from Central Location ?

[G0]: This is an open question for me also.


Several Glances with an independent synchronization service(PUSH)

  *   I refer to this as the PUSH model
  *   I don’t believe you have to ( or necessarily should) rely on the backend 
to do the synchronization of the images
 *   i.e. the ‘Synch Service’ could do this strictly through Glance REST 
APIs
(making it independent of the particular Glance backend ... and allowing the 
Glance Backends at Central and Edge sites to actually be different)
[G0]: Okay, I can update the wiki to reflect this. Should we keep the 
“synchronization by the backend” option as an other alternative?
[Greg] Yeah we should keep it as an alternative.

  *   I think the ‘Synch Service’ MUST be able to support ‘selective/multicast’ 
distribution of Images from Central to Edge for Image Synchronization
 *   i.e. you don’t want Central Site pushing ALL images to ALL Edge Sites 
... especially for the small Edge Sites
[G0]: Yes, the question is how to define these synchronization policies.
[Greg] Agreed ... we’ve had some very high-level discussions with end users, 
but haven’t put together a proposal yet.

  *   Not sure ... but I didn’t think this was the model being used in mixmatch 
... thought mixmatch was more the PULL model (below)

[G0]: Yes, this is more or less my understanding. I remove the mixmatch 
reference from this chapter.

One Glance and multiple Glance API Servers   (PULL)

  *   I refer to this as the PULL model
  *   This is the current model supported in StarlingX’s Distributed Cloud 
sub-project
 *   We run glance-api on all Edge Clouds ... that talk to glance-registry 
on the Central Cloud, and
 *   We have glance-api setup for caching such that only the first access 
to an particular image incurs the latency of the image transfer from Central to 
Edge
[G0]: Do you do image caching in Glance API or do you rely in the image cache 
in Nova? In the Forum session there were some discussions about this and I 
think the conclusion was that using the image cache of Nova is enough.
[Greg] We enabled image caching in the Glance API.
 I believe that Nova Image Caching caches at the compute node ... 
this would work ok for all-in-one edge clouds or small edge clouds.
 But glance-api caching caches at the edge cloud level, so works 
better for large edge clouds with lots of compute nodes.

  *
  *   this PULL model affectively implements the location aware synchronization 
you talk about below,  (i.e. synchronise images only to those cloud instances 
where they are needed)?


In StarlingX Distributed Cloud,
We plan on supporting both the PUSH and PULL model ... suspect there are use 
cases for both.

[G0]: This means that you need an architecture supporting both.
Just for my curiosity what is the use case for the pull model once you have the 
push model in place?
[Greg] The PULL model certainly results in the most efficient distribution of 
images ... basically images are distributed ONLY to edge clouds that explicitly 
use the image.
Also if the use case is NOT concerned about incurring the latency of the image 
transfer from Central to Edge on the FIRST use of image then the PULL model 
could be preferred ... TBD.

Here is the updated wiki: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment
[Greg] Looks good.

Greg.


Thanks,
Gerg0




From: "Csatari, Gergely (Nokia - HU/Budapest)" 
mailto:gergely.csat...@nokia.com>>
Date: Thursday, June 7, 2018 at 6:49 AM
To: 
"openstack-dev@lists.openstack.org" 
mailto:openstack-dev@lists.openstack.org>>, 
"edge-comput...@lists.openstack.org" 
mailto:edge-comput...@lists.openstack.org>>
Subject: Re: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

I did some work ont he figures and realised, that I have some questions related 
to the alternative options:

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Csatari, Gergely (Nokia - HU/Budapest)]

Hi,

Going inline.

From: Waines, Greg [mailto:greg.wai...@windriver.com]
Sent: Thursday, June 7, 2018 2:24 PM

I had some additional questions/comments on the Image Synchronization Options ( 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment ):


One Glance with multiple backends

  *   In this scenario, are all Edge Clouds simply configured with the one 
central glance for its GLANCE ENDPOINT ?
 *   i.e. GLANCE is a typical shared service in a multi-region environment ?

[G0]: In my understanding yes.


  *   If so,
how does this OPTION support the requirement for Edge Cloud Operation when 
disconnected from Central Location ?

[G0]: This is an open question for me also.


Several Glances with an independent synchronization service(PUSH)

  *   I refer to this as the PUSH model
  *   I don’t believe you have to ( or necessarily should) rely on the backend 
to do the synchronization of the images
 *   i.e. the ‘Synch Service’ could do this strictly through Glance REST 
APIs
(making it independent of the particular Glance backend ... and allowing the 
Glance Backends at Central and Edge sites to actually be different)
[G0]: Okay, I can update the wiki to reflect this. Should we keep the 
“synchronization by the backend” option as an other alternative?

  *   I think the ‘Synch Service’ MUST be able to support ‘selective/multicast’ 
distribution of Images from Central to Edge for Image Synchronization
 *   i.e. you don’t want Central Site pushing ALL images to ALL Edge Sites 
... especially for the small Edge Sites
[G0]: Yes, the question is how to define these synchronization policies.

  *   Not sure ... but I didn’t think this was the model being used in mixmatch 
... thought mixmatch was more the PULL model (below)

[G0]: Yes, this is more or less my understanding. I remove the mixmatch 
reference from this chapter.

One Glance and multiple Glance API Servers   (PULL)

  *   I refer to this as the PULL model
  *   This is the current model supported in StarlingX’s Distributed Cloud 
sub-project
 *   We run glance-api on all Edge Clouds ... that talk to glance-registry 
on the Central Cloud, and
 *   We have glance-api setup for caching such that only the first access 
to an particular image incurs the latency of the image transfer from Central to 
Edge
[G0]: Do you do image caching in Glance API or do you rely in the image cache 
in Nova? In the Forum session there were some discussions about this and I 
think the conclusion was that using the image cache of Nova is enough.

  *
  *   this PULL model affectively implements the location aware synchronization 
you talk about below,  (i.e. synchronise images only to those cloud instances 
where they are needed)?


In StarlingX Distributed Cloud,
We plan on supporting both the PUSH and PULL model ... suspect there are use 
cases for both.

[G0]: This means that you need an architecture supporting both.
Just for my curiosity what is the use case for the pull model once you have the 
push model in place?

Here is the updated wiki: 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment

Thanks,
Gerg0




From: "Csatari, Gergely (Nokia - HU/Budapest)" 
mailto:gergely.csat...@nokia.com>>
Date: Thursday, June 7, 2018 at 6:49 AM
To: 
"openstack-dev@lists.openstack.org" 
mailto:openstack-dev@lists.openstack.org>>, 
"edge-comput...@lists.openstack.org" 
mailto:edge-comput...@lists.openstack.org>>
Subject: Re: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

I did some work ont he figures and realised, that I have some questions related 
to the alternative options:

Multiple backends option:

  *   What is the API between Glance and the Glance backends?
  *   How is it possible to implement location aware synchronisation 
(synchronise images only to those cloud instances where they are needed)?
  *   Is it possible to have different OpenStack versions in the different 
cloud instances?
  *   Can a cloud instance use the locally synchronised images in case of a 
network connection break?
  *   Is it possible to implement this without storing database credentials ont 
he edge cloud instances?

Independent synchronisation service:

  *   If I understood [1] correctly 
mixmatch can help Nova to attach a remote volume, but it will not help in 
synchronizing the images. is this true?


As I promised in the Edge Compute Group call I plan to organize an IRC review 
meeting to check the wiki. Please indicate your availability in 
[2].

[1]: https://mixmatch.readthedocs.io/en/latest/
[2]: https://doodle.com/poll/bddg65vyh4qwxpk5

Br,
Gerg0

From: Csatari, Gergely (Nokia - HU/Budapest)
Sent: Wednesday, May 23, 2018 8:59 PM
To: OpenStack Development Mailing List (not for usage questions) 

Re: [openstack-dev] [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible architectures for image synchronisation

[Waines, Greg]

I had some additional questions/comments on the Image Synchronization Options ( 
https://wiki.openstack.org/wiki/Image_handling_in_edge_environment ):


One Glance with multiple backends

· In this scenario, are all Edge Clouds simply configured with the one 
central glance for its GLANCE ENDPOINT ?

oi.e. GLANCE is a typical shared service in a multi-region environment ?

· If so,
how does this OPTION support the requirement for Edge Cloud Operation when 
disconnected from Central Location ?



Several Glances with an independent synchronization service(PUSH)

· I refer to this as the PUSH model

· I don’t believe you have to ( or necessarily should) rely on the 
backend to do the synchronization of the images

oi.e. the ‘Synch Service’ could do this strictly through Glance REST APIs
(making it independent of the particular Glance backend ... and allowing the 
Glance Backends at Central and Edge sites to actually be different)


· I think the ‘Synch Service’ MUST be able to support 
‘selective/multicast’ distribution of Images from Central to Edge for Image 
Synchronization

oi.e. you don’t want Central Site pushing ALL images to ALL Edge Sites ... 
especially for the small Edge Sites


· Not sure ... but I didn’t think this was the model being used in 
mixmatch ... thought mixmatch was more the PULL model (below)



One Glance and multiple Glance API Servers   (PULL)

· I refer to this as the PULL model

· This is the current model supported in StarlingX’s Distributed Cloud 
sub-project

oWe run glance-api on all Edge Clouds ... that talk to glance-registry on 
the Central Cloud, and

oWe have glance-api setup for caching such that only the first access to an 
particular image incurs the latency of the image transfer from Central to Edge

·

· this PULL model affectively implements the location aware 
synchronization you talk about below,  (i.e. synchronise images only to those 
cloud instances where they are needed)?


In StarlingX Distributed Cloud,
We plan on supporting both the PUSH and PULL model ... suspect there are use 
cases for both.

Greg.





From: "Csatari, Gergely (Nokia - HU/Budapest)" 
Date: Thursday, June 7, 2018 at 6:49 AM
To: "openstack-dev@lists.openstack.org" , 
"edge-comput...@lists.openstack.org" 
Subject: Re: [Edge-computing] [edge][glance][mixmatch]: Wiki of the possible 
architectures for image synchronisation

Hi,

I did some work ont he figures and realised, that I have some questions related 
to the alternative options:

Multiple backends option:
-  What is the API between Glance and the Glance backends?
-  How is it possible to implement location aware synchronisation 
(synchronise images only to those cloud instances where they are needed)?
-  Is it possible to have different OpenStack versions in the different 
cloud instances?
-  Can a cloud instance use the locally synchronised images in case of 
a network connection break?
-  Is it possible to implement this without storing database 
credentials ont he edge cloud instances?

Independent synchronisation service:
-  If I understood [1] 
correctly mixmatch can help Nova to attach a remote volume, but it will not 
help in synchronizing the images. is this true?


As I promised in the Edge Compute Group call I plan to organize an IRC review 
meeting to check the wiki. Please indicate your availability in 
[2].

[1]: https://mixmatch.readthedocs.io/en/latest/
[2]: https://doodle.com/poll/bddg65vyh4qwxpk5

Br,
Gerg0

From: Csatari, Gergely (Nokia - HU/Budapest)
Sent: Wednesday, May 23, 2018 8:59 PM
To: OpenStack Development Mailing List (not for usage questions) 
; edge-comput...@lists.openstack.org
Subject: [edge][glance]: Wiki of the possible architectures for image 
synchronisation

Hi,

Here I send the wiki page 
[1] where I 
summarize what I understood from the Forum session about image synchronisation 
in edge environment [2], [3].

Please check and correct/comment.

Thanks,
Gerg0


[1]: https://wiki.openstack.org/wiki/Image_handling_in_edge_environment
[2]: https://etherpad.openstack.org/p/yvr-edge-cloud-images
[3]: 
https://www.openstack.org/summit/vancouver-2018/summit-schedule/events/21768/image-handling-in-an-edge-cloud-infrastructure
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev