Re: [openstack-dev] [nova][cinder] Is there interest in an admin-api to refresh volume connection info?

2017-06-08 Thread Matt Riedemann

On 6/8/2017 6:17 PM, John Griffith wrote:
​The attachment_update call could do this for you, might need some 
slight tweaks because I tried to make sure that we weren't having 
attachment records be modified things that lived forever and were 
dynamic.  This particular case seems like a descent fit though, issue 
the call; cinder queries the backend to get any updated connection info 
and sends it back.  I'd leave it to Nova to figure out if said info has 
been updated or not.  Just iterate through the attachment_ids in the bdm 
and update/refresh each one maybe?


Yeah, although we have to keep in mind that's a new API we're not even 
using yet for volume attach, so anything I'm thinking about here has to 
handle old-style attachments (old-style as in, you know, today). Plus we 
don't have a migration plan yet for the old style attachments to the new 
style. At the Pike PTG we said we'd work on that in Queens.


I definitely want to use new shiny things at some point, we just have to 
handle the old crufty things too.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder] Is there interest in an admin-api to refresh volume connection info?

2017-06-08 Thread John Griffith
On Thu, Jun 8, 2017 at 7:58 AM, Matt Riedemann  wrote:

> Nova stores the output of the Cinder os-initialize_connection info API in
> the Nova block_device_mappings table, and uses that later for making volume
> connections.
>
> This data can get out of whack or need to be refreshed, like if your ceph
> server IP changes, or you need to recycle some secret uuid for your ceph
> cluster.
>
> I think the only ways to do this on the nova side today are via volume
> detach/re-attach, reboot, migrations, etc - all of which, except live
> migration, are disruptive to the running guest.
>
> I've kicked around the idea of adding some sort of admin API interface for
> refreshing the BDM.connection_info on-demand if needed by an operator. Does
> anyone see value in this? Are operators doing stuff like this already, but
> maybe via direct DB updates?
>
> We could have something in the compute API which calls down to the compute
> for an instance and has it refresh the connection_info from Cinder and
> updates the BDM table in the nova DB. It could be an admin action API, or
> part of the os-server-external-events API, like what we have for the
> 'network-changed' event sent from Neutron which nova uses to refresh the
> network info cache.
>
> Other ideas or feedback here?
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
​The attachment_update call could do this for you, might need some slight
tweaks because I tried to make sure that we weren't having attachment
records be modified things that lived forever and were dynamic.  This
particular case seems like a descent fit though, issue the call; cinder
queries the backend to get any updated connection info and sends it back.
I'd leave it to Nova to figure out if said info has been updated or not.
Just iterate through the attachment_ids in the bdm and update/refresh each
one maybe?
​
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder] Is there interest in an admin-api to refresh volume connection info?

2017-06-08 Thread Arne Wiebalck

> On 08 Jun 2017, at 17:52, Matt Riedemann  wrote:
> 
> On 6/8/2017 10:17 AM, Arne Wiebalck wrote:
>>> On 08 Jun 2017, at 15:58, Matt Riedemann >> > wrote:
>>> 
>>> Nova stores the output of the Cinder os-initialize_connection info API in 
>>> the Nova block_device_mappings table, and uses that later for making volume 
>>> connections.
>>> 
>>> This data can get out of whack or need to be refreshed, like if your ceph 
>>> server IP changes, or you need to recycle some secret uuid for your ceph 
>>> cluster.
>>> 
>>> I think the only ways to do this on the nova side today are via volume 
>>> detach/re-attach, reboot, migrations, etc - all of which, except live 
>>> migration, are disruptive to the running guest.
>>> 
>>> I've kicked around the idea of adding some sort of admin API interface for 
>>> refreshing the BDM.connection_info on-demand if needed by an operator. Does 
>>> anyone see value in this? Are operators doing stuff like this already, but 
>>> maybe via direct DB updates?
>>> 
>>> We could have something in the compute API which calls down to the compute 
>>> for an instance and has it refresh the connection_info from Cinder and 
>>> updates the BDM table in the nova DB. It could be an admin action API, or 
>>> part of the os-server-external-events API, like what we have for the 
>>> 'network-changed' event sent from Neutron which nova uses to refresh the 
>>> network info cache.
>>> 
>>> Other ideas or feedback here?
>> I have opened https://bugs.launchpad.net/cinder/+bug/1452641 for this issue 
>> some time ago.
>> Back then I was more thinking of using an alias and not deal with IP 
>> addresses directly. From
>> what I understand, this should work with Ceph. In any case, there is still 
>> interest in a fix :-)
>> Cheers,
>>  Arne
>> --
>> Arne Wiebalck
>> CERN IT
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> Yeah this was also discussed in the dev mailing list over a year ago:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2016-May/095170.html 
> 
> 
> At that time I was opposed to a REST API for a *user* doing this, but I'm 
> more open to an *admin* (by default) doing this. Also, if it were initiated 
> via the volume API then Cinder could call the Nova os-server-external-events 
> API which is admin-only by default and then Nova can do a refresh.
> 
> Later in that thread Melanie Witt also has an idea about doing a refresh in a 
> periodic task on the compute service, like we do for refreshing the instance 
> network info cache with Neutron in a periodic task.

Wouldn’t using a mon alias (and not resolving it to the respective IP 
addresses) be enough? Or is that too backend specific?

The idea of a periodic task leveraging existing techniques sounds really nice, 
but if the overhead is regarded as too much (in the end, the IP addresses 
shouldn’t change that often), an admin only API to be called when the addresses 
need to be updated sounds good to me as well.

Cheers,
 Arne

—
Arne Wiebalck
CERN IT__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder] Is there interest in an admin-api to refresh volume connection info?

2017-06-08 Thread Matt Riedemann

On 6/8/2017 10:17 AM, Arne Wiebalck wrote:


On 08 Jun 2017, at 15:58, Matt Riedemann > wrote:


Nova stores the output of the Cinder os-initialize_connection info API 
in the Nova block_device_mappings table, and uses that later for 
making volume connections.


This data can get out of whack or need to be refreshed, like if your 
ceph server IP changes, or you need to recycle some secret uuid for 
your ceph cluster.


I think the only ways to do this on the nova side today are via volume 
detach/re-attach, reboot, migrations, etc - all of which, except live 
migration, are disruptive to the running guest.


I've kicked around the idea of adding some sort of admin API interface 
for refreshing the BDM.connection_info on-demand if needed by an 
operator. Does anyone see value in this? Are operators doing stuff 
like this already, but maybe via direct DB updates?


We could have something in the compute API which calls down to the 
compute for an instance and has it refresh the connection_info from 
Cinder and updates the BDM table in the nova DB. It could be an admin 
action API, or part of the os-server-external-events API, like what we 
have for the 'network-changed' event sent from Neutron which nova uses 
to refresh the network info cache.


Other ideas or feedback here?


I have opened https://bugs.launchpad.net/cinder/+bug/1452641 for this 
issue some time ago.
Back then I was more thinking of using an alias and not deal with IP 
addresses directly. From
what I understand, this should work with Ceph. In any case, there is 
still interest in a fix :-)


Cheers,
  Arne


--
Arne Wiebalck
CERN IT



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Yeah this was also discussed in the dev mailing list over a year ago:

http://lists.openstack.org/pipermail/openstack-dev/2016-May/095170.html

At that time I was opposed to a REST API for a *user* doing this, but 
I'm more open to an *admin* (by default) doing this. Also, if it were 
initiated via the volume API then Cinder could call the Nova 
os-server-external-events API which is admin-only by default and then 
Nova can do a refresh.


Later in that thread Melanie Witt also has an idea about doing a refresh 
in a periodic task on the compute service, like we do for refreshing the 
instance network info cache with Neutron in a periodic task.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder] Is there interest in an admin-api to refresh volume connection info?

2017-06-08 Thread Arne Wiebalck

On 08 Jun 2017, at 15:58, Matt Riedemann 
> wrote:

Nova stores the output of the Cinder os-initialize_connection info API in the 
Nova block_device_mappings table, and uses that later for making volume 
connections.

This data can get out of whack or need to be refreshed, like if your ceph 
server IP changes, or you need to recycle some secret uuid for your ceph 
cluster.

I think the only ways to do this on the nova side today are via volume 
detach/re-attach, reboot, migrations, etc - all of which, except live 
migration, are disruptive to the running guest.

I've kicked around the idea of adding some sort of admin API interface for 
refreshing the BDM.connection_info on-demand if needed by an operator. Does 
anyone see value in this? Are operators doing stuff like this already, but 
maybe via direct DB updates?

We could have something in the compute API which calls down to the compute for 
an instance and has it refresh the connection_info from Cinder and updates the 
BDM table in the nova DB. It could be an admin action API, or part of the 
os-server-external-events API, like what we have for the 'network-changed' 
event sent from Neutron which nova uses to refresh the network info cache.

Other ideas or feedback here?

I have opened https://bugs.launchpad.net/cinder/+bug/1452641 for this issue 
some time ago.
Back then I was more thinking of using an alias and not deal with IP addresses 
directly. From
what I understand, this should work with Ceph. In any case, there is still 
interest in a fix :-)

Cheers,
 Arne


--
Arne Wiebalck
CERN IT

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev