Re: [Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Jay Pipes]

On 07/06/2017 02:17 AM, Eddie Yen wrote:

Hi, now I got another problem.

I have two models of GPU devices and I set both in pci_alias and 
pcli_passthrogh_whitelist on controller and compute node(with this two 
GPUs).
Now I removed one of GPU and delete its data in nova.conf, then restart 
nova-api, nova-scheduler (Controller) and nova-compute(Compute)


But when I check MySQL, the GPU info which I already removed still in 
pci_devices table.

I remember there's a bug about this case but it already fixed.

How can I fix this issue?


Eddie, please start a new mailing list thread (with a new targeted 
topic) for the above and we'll answer it there to keep the mailing list 
threads properly curated.


Best,
-jay

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Eddie Yen]

Hi, now I got another problem.

I have two models of GPU devices and I set both in pci_alias and
pcli_passthrogh_whitelist on controller and compute node(with this two
GPUs).
Now I removed one of GPU and delete its data in nova.conf, then restart
nova-api, nova-scheduler (Controller) and nova-compute(Compute)

But when I check MySQL, the GPU info which I already removed still in
pci_devices table.
I remember there's a bug about this case but it already fixed.

How can I fix this issue?

Thanks,


2017-07-06 11:42 GMT+08:00 Eddie Yen :

> Thanks Kevin,
>
> This is another good option to solve IP problem for me. I'll note this.
>
> Eddie.
>
> 2017-07-06 11:10 GMT+08:00 Kevin Benton :
>
>> As a stop gap for the IP problem, keep in mind that you can create a
>> neutron port separately and boot the instance to it. Then when you delete
>> the instance the port will remain and hold onto the IP so you can boot to
>> it with a different instance later.
>>
>> On Jul 5, 2017 18:31, "Eddie Yen"  wrote:
>>
>>> Hi, thanks for reply first!
>>>
>>> After I tried shelve instance, yes it really release GPU resources.
>>> But as Dinesh said, can't unshelve this instance if all GPUs occupied.
>>>
>>> Before known about shelve instance, I have do few approach.
>>>
>>> One is using "resize" the instance's flavor to non-GPU flavor, but can't
>>> do this if all GPUs occupied.
>>>
>>> Another one is like Jay's recommend is that create instance as volume,
>>> and delete instance but keep volume if this VM not going to use GPU.
>>> Then re-create instance with previous volume and non-GPU flavor.
>>> And yes, this approach have IP address problem that may cause
>>> application atypical.
>>>
>>> I'll try how to make my application not to rely on a persistent IP.
>>>
>>> Thanks,
>>> Eddie.
>>>
>>> 2017-07-05 19:36 GMT+08:00 Jay Pipes :
>>>
 On 07/05/2017 04:18 AM, Eddie Yen wrote:

> Hi everyone,
>
> I'm using OpenStack Mitaka (which deployed from Fuel 9.2) and doing
> GPU things.
>
> I got a problem is I need to delete current instance which contains
> GPU to release device if I want assign GPU to another new instance
> temperately.
>
> I'll got "No valid host was found" if I creating new instance contains
> GPU flavor without delete present instance, even the instance is shutdown.
>
> Is there any way that I just shutdown the instance rather than delete
> it to release GPU device?
>

 As Dinesh mentioned you *can* use shelve for this, but frankly, I think
 the shelve API leads to more problems than it solves (see his response
 about needing to delete the new instance before unshelving).

 I'd recommend redesigning your application to be more cloud-native. In
 other words, separate operating system state from application state, use
 volumes for all persistent application state, and do not rely on a
 persistent IP address. [1]

 Once you've done that, you can just treat your VMs like cattle and
 terminate them.

 Best,
 -jay

 [1] Please note I did not use the word "container" in this description
 of cloud-native application.

 ___
 Mailing list: http://lists.openstack.org/cgi
 -bin/mailman/listinfo/openstack
 Post to : openstack@lists.openstack.org
 Unsubscribe : http://lists.openstack.org/cgi
 -bin/mailman/listinfo/openstack

>>>
>>>
>>> ___
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>>
>>>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Eddie Yen]

Thanks Kevin,

This is another good option to solve IP problem for me. I'll note this.

Eddie.

2017-07-06 11:10 GMT+08:00 Kevin Benton :

> As a stop gap for the IP problem, keep in mind that you can create a
> neutron port separately and boot the instance to it. Then when you delete
> the instance the port will remain and hold onto the IP so you can boot to
> it with a different instance later.
>
> On Jul 5, 2017 18:31, "Eddie Yen"  wrote:
>
>> Hi, thanks for reply first!
>>
>> After I tried shelve instance, yes it really release GPU resources.
>> But as Dinesh said, can't unshelve this instance if all GPUs occupied.
>>
>> Before known about shelve instance, I have do few approach.
>>
>> One is using "resize" the instance's flavor to non-GPU flavor, but can't
>> do this if all GPUs occupied.
>>
>> Another one is like Jay's recommend is that create instance as volume,
>> and delete instance but keep volume if this VM not going to use GPU.
>> Then re-create instance with previous volume and non-GPU flavor.
>> And yes, this approach have IP address problem that may cause application
>> atypical.
>>
>> I'll try how to make my application not to rely on a persistent IP.
>>
>> Thanks,
>> Eddie.
>>
>> 2017-07-05 19:36 GMT+08:00 Jay Pipes :
>>
>>> On 07/05/2017 04:18 AM, Eddie Yen wrote:
>>>
 Hi everyone,

 I'm using OpenStack Mitaka (which deployed from Fuel 9.2) and doing GPU
 things.

 I got a problem is I need to delete current instance which contains GPU
 to release device if I want assign GPU to another new instance temperately.

 I'll got "No valid host was found" if I creating new instance contains
 GPU flavor without delete present instance, even the instance is shutdown.

 Is there any way that I just shutdown the instance rather than delete
 it to release GPU device?

>>>
>>> As Dinesh mentioned you *can* use shelve for this, but frankly, I think
>>> the shelve API leads to more problems than it solves (see his response
>>> about needing to delete the new instance before unshelving).
>>>
>>> I'd recommend redesigning your application to be more cloud-native. In
>>> other words, separate operating system state from application state, use
>>> volumes for all persistent application state, and do not rely on a
>>> persistent IP address. [1]
>>>
>>> Once you've done that, you can just treat your VMs like cattle and
>>> terminate them.
>>>
>>> Best,
>>> -jay
>>>
>>> [1] Please note I did not use the word "container" in this description
>>> of cloud-native application.
>>>
>>> ___
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>>
>>
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Kevin Benton]

As a stop gap for the IP problem, keep in mind that you can create a
neutron port separately and boot the instance to it. Then when you delete
the instance the port will remain and hold onto the IP so you can boot to
it with a different instance later.

On Jul 5, 2017 18:31, "Eddie Yen"  wrote:

> Hi, thanks for reply first!
>
> After I tried shelve instance, yes it really release GPU resources.
> But as Dinesh said, can't unshelve this instance if all GPUs occupied.
>
> Before known about shelve instance, I have do few approach.
>
> One is using "resize" the instance's flavor to non-GPU flavor, but can't
> do this if all GPUs occupied.
>
> Another one is like Jay's recommend is that create instance as volume, and
> delete instance but keep volume if this VM not going to use GPU.
> Then re-create instance with previous volume and non-GPU flavor.
> And yes, this approach have IP address problem that may cause application
> atypical.
>
> I'll try how to make my application not to rely on a persistent IP.
>
> Thanks,
> Eddie.
>
> 2017-07-05 19:36 GMT+08:00 Jay Pipes :
>
>> On 07/05/2017 04:18 AM, Eddie Yen wrote:
>>
>>> Hi everyone,
>>>
>>> I'm using OpenStack Mitaka (which deployed from Fuel 9.2) and doing GPU
>>> things.
>>>
>>> I got a problem is I need to delete current instance which contains GPU
>>> to release device if I want assign GPU to another new instance temperately.
>>>
>>> I'll got "No valid host was found" if I creating new instance contains
>>> GPU flavor without delete present instance, even the instance is shutdown.
>>>
>>> Is there any way that I just shutdown the instance rather than delete it
>>> to release GPU device?
>>>
>>
>> As Dinesh mentioned you *can* use shelve for this, but frankly, I think
>> the shelve API leads to more problems than it solves (see his response
>> about needing to delete the new instance before unshelving).
>>
>> I'd recommend redesigning your application to be more cloud-native. In
>> other words, separate operating system state from application state, use
>> volumes for all persistent application state, and do not rely on a
>> persistent IP address. [1]
>>
>> Once you've done that, you can just treat your VMs like cattle and
>> terminate them.
>>
>> Best,
>> -jay
>>
>> [1] Please note I did not use the word "container" in this description of
>> cloud-native application.
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Eddie Yen]

Hi, thanks for reply first!

After I tried shelve instance, yes it really release GPU resources.
But as Dinesh said, can't unshelve this instance if all GPUs occupied.

Before known about shelve instance, I have do few approach.

One is using "resize" the instance's flavor to non-GPU flavor, but can't do
this if all GPUs occupied.

Another one is like Jay's recommend is that create instance as volume, and
delete instance but keep volume if this VM not going to use GPU.
Then re-create instance with previous volume and non-GPU flavor.
And yes, this approach have IP address problem that may cause application
atypical.

I'll try how to make my application not to rely on a persistent IP.

Thanks,
Eddie.

2017-07-05 19:36 GMT+08:00 Jay Pipes :

> On 07/05/2017 04:18 AM, Eddie Yen wrote:
>
>> Hi everyone,
>>
>> I'm using OpenStack Mitaka (which deployed from Fuel 9.2) and doing GPU
>> things.
>>
>> I got a problem is I need to delete current instance which contains GPU
>> to release device if I want assign GPU to another new instance temperately.
>>
>> I'll got "No valid host was found" if I creating new instance contains
>> GPU flavor without delete present instance, even the instance is shutdown.
>>
>> Is there any way that I just shutdown the instance rather than delete it
>> to release GPU device?
>>
>
> As Dinesh mentioned you *can* use shelve for this, but frankly, I think
> the shelve API leads to more problems than it solves (see his response
> about needing to delete the new instance before unshelving).
>
> I'd recommend redesigning your application to be more cloud-native. In
> other words, separate operating system state from application state, use
> volumes for all persistent application state, and do not rely on a
> persistent IP address. [1]
>
> Once you've done that, you can just treat your VMs like cattle and
> terminate them.
>
> Best,
> -jay
>
> [1] Please note I did not use the word "container" in this description of
> cloud-native application.
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Jay Pipes]

On 07/05/2017 04:18 AM, Eddie Yen wrote:

Hi everyone,

I'm using OpenStack Mitaka (which deployed from Fuel 9.2) and doing GPU 
things.


I got a problem is I need to delete current instance which contains GPU 
to release device if I want assign GPU to another new instance temperately.


I'll got "No valid host was found" if I creating new instance contains 
GPU flavor without delete present instance, even the instance is shutdown.


Is there any way that I just shutdown the instance rather than delete it 
to release GPU device?


As Dinesh mentioned you *can* use shelve for this, but frankly, I think 
the shelve API leads to more problems than it solves (see his response 
about needing to delete the new instance before unshelving).


I'd recommend redesigning your application to be more cloud-native. In 
other words, separate operating system state from application state, use 
volumes for all persistent application state, and do not rely on a 
persistent IP address. [1]


Once you've done that, you can just treat your VMs like cattle and 
terminate them.


Best,
-jay

[1] Please note I did not use the word "container" in this description 
of cloud-native application.


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Bhor, Dinesh]

Hi Eddie,

One way to do this is to “shelve” the instance which will allow to use your PCI 
device for other instance.
But you won’t be able to “unshelve” the previous instance until you delete or 
unshelve the newly created instance.

NOTE:
Shelving will release all the resources associated to that instance if the 
config option shelved_offload_time is not -1.

Thanks and Regards,
Dinesh Bhor | App. Software Dev. Cnslt.
dinesh.b...@nttdata.com<mailto:dinesh.b...@nttdata.com> | VOIP. 8833.8395I | 
nttdata.com/americas<http://nttdata.com/americas>
NTT DATA, Inc.
Consulting | Digital | Managed Services | Industry Solutions
Learn more:
[Description: Description: 
cid:image005.jpg@01D193F0.F70B44C0]<http://americas.nttdata.com/>

[Description: Description: 
cid:image009.jpg@01D193F0.F70B44C0]<http://www.linkedin.com/company/ntt-data-americas>

[Description: Description: 
cid:image010.jpg@01D193F0.F70B44C0]<https://twitter.com/NTTDATAAmericas>

[Description: Description: 
cid:image011.jpg@01D193F0.F70B44C0]<https://www.facebook.com/NTTDATAAmericas>



From: Eddie Yen [mailto:missile0...@gmail.com]
Sent: Wednesday, July 05, 2017 1:49 PM
To: openstack@lists.openstack.org
Subject: [Openstack] [OpenStack] [nova] How can I release PCI device resources 
without delete instance?

Hi everyone,

I'm using OpenStack Mitaka (which deployed from Fuel 9.2) and doing GPU things.

I got a problem is I need to delete current instance which contains GPU to 
release device if I want assign GPU to another new instance temperately.

I'll got "No valid host was found" if I creating new instance contains GPU 
flavor without delete present instance, even the instance is shutdown.

Is there any way that I just shutdown the instance rather than delete it to 
release GPU device?


Many thanks,
Eddie.

__
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] [OpenStack] [nova] How can I release PCI device resources without delete instance?

[Eddie Yen]

Hi everyone,

I'm using OpenStack Mitaka (which deployed from Fuel 9.2) and doing GPU
things.

I got a problem is I need to delete current instance which contains GPU to
release device if I want assign GPU to another new instance temperately.

I'll got "No valid host was found" if I creating new instance contains GPU
flavor without delete present instance, even the instance is shutdown.

Is there any way that I just shutdown the instance rather than delete it to
release GPU device?


Many thanks,
Eddie.
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack