[Openstack-operators] Avoiding Cinder stampedes with RBD? (Kilo)

[Ned Rhudy (BLOOMBERG/ 731 LEX)]

We have a situation where tenant A is trying to launch large numbers of 
instances from a single RBD volume snapshot in Cinder (e.g., 40 instances at 
once). We made an unrelated change recently to enable 
rbd_flatten_volume_from_snapshot by default in order to save tenants who create 
large chains of volumes and snapshots until they run out of quota and then 
can't figure out how to unwind the chain, because the relationships are not 
trivially traceable. This change appears to be causing tenant A significant 
heartache now, because when he launches some large number of instances at once, 
most of his instance launches time out, presumably because Cinder is stampeding 
onto Ceph and trying to create flattened RBD images for every single instance 
simultaneously.

My question is, is there anything I can do to stop this stampeding? A review of 
cinder.conf options for Kilo didn't point out any obvious setting that could be 
adjusted to force Cinder to serialize its operations here, but maybe I missed 
something.___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [nova] create image from paused VM refused with http 409

[Saverio Proto]

Hello there,

I used to do this in Juno, and now I upgraded to Kilo and it is not
working anymore.

macsp:~ proto$ openstack server image create --name test
81da19c6-efbe-4002-b4e8-5ce352ffdd14
Cannot 'createImage' instance 81da19c6-efbe-4002-b4e8-5ce352ffdd14
while it is in vm_state paused (HTTP 409) (Request-ID:
req-0e7f339d-a236-4584-a44c-49daed7558ee)

Is this a change of behaviour ?

I also found this:

https://review.openstack.org/#/c/116789/

Should not this work ? my nova-api --version is 2015.1.2

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [osops] OSOps Tools and Monitoring Meeting - Wendesday 1900 UTC

[Joseph Bajin]

Operators,


We will be having our bi-weekly meeting in the #openstack-meeting-4 room at
1900UTC.

I have published the current agenda for this meeting. You can find that
here:

https://etherpad.openstack.org/p/osops-irc-meeting-20160323


Either let me know or please go ahead and add to the ether pad anything
else that you would like to discuss during the meeting.


Thanks


Joe
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] create image from paused VM refused with http 409

[Christopher Hull]

I had some interesting experiences between Cinder and Glance with Kilo as
well.  For instance, occasionally I'd get a bad checksum when running an
image created from a Cinder volume.  The checksum would occasionally be
wrong.   Liberty works well.   All my images run no matter what.   But I
can't get a Liberty based Neutron Router to work.  The Public Port is
always Down.  Resorting to dedicating a small instance to be my virtual
router.

Best of luck.  If your problem keeps up upgrade to Liberty.
Chris

- Christopher T. Hull
I am presently seeking a new career opportunity  Please see career page
http://chrishull.com/career
333 Orchard Ave, Sunnyvale CA. 94085
(415) 385 4865
chrishul...@gmail.com
http://chrishull.com



On Tue, Mar 22, 2016 at 7:36 AM, Saverio Proto  wrote:

> I am not a developer but I tried my best !
> https://review.openstack.org/#/c/295865/
>
> I applied the patch in my staging system and it fixes the problem :)
>
> Saverio
>
>
> 2016-03-22 13:46 GMT+01:00 Saverio Proto :
> > I found the problem. It happens only then the instance has booted from
> volume.
> >
> > I think api.py should be patched at the function
> > snapshot_volume_backed (line 2250) in a similar way as
> > https://review.openstack.org/#/c/116789/
> >
> > Looks like the bug is also there in master , file nova/compute/api.py
> line 2296
> >
> > I will try to submit a patch with Gerrit
> >
> > Saverio
> >
> > 2016-03-22 13:33 GMT+01:00 Saverio Proto :
> >> Hello there,
> >>
> >> I used to do this in Juno, and now I upgraded to Kilo and it is not
> >> working anymore.
> >>
> >> macsp:~ proto$ openstack server image create --name test
> >> 81da19c6-efbe-4002-b4e8-5ce352ffdd14
> >> Cannot 'createImage' instance 81da19c6-efbe-4002-b4e8-5ce352ffdd14
> >> while it is in vm_state paused (HTTP 409) (Request-ID:
> >> req-0e7f339d-a236-4584-a44c-49daed7558ee)
> >>
> >> Is this a change of behaviour ?
> >>
> >> I also found this:
> >>
> >> https://review.openstack.org/#/c/116789/
> >>
> >> Should not this work ? my nova-api --version is 2015.1.2
> >>
> >> Saverio
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [tags] Appealing for change of tags

[Tim Bell]

Looking at projects like Barbican in the 
https://www.openstack.org/software/releases/liberty/components/barbican, they 
seem to have met some conditions which are currently marked as ‘no’.

What is the mechanism to query these settings (either manual or automatic) ?

As an example, there is significant documentation for Barbican at  
http://docs.openstack.org/developer/barbican/ but it is marked as ‘no’ in the 
page for 
https://www.openstack.org/software/releases/liberty/components/barbican  Is 
there an install guide for this project guide (at docs.openstack.org)?

How about adding

  *   some algorithmic description (i.e. automatic or manual) for the selection 
? If automatic, the algorithm could be provided so the project knows what to do
  *   a process description for the manual cases for projects to ask for a 
re-classification

Tim

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [tags] Appealing for change of tags

[Gal Sagie]

The projects themselves don't seems updated, there are other projects in
the big-tent that are not
listed there yet.

On Tue, Mar 22, 2016 at 9:16 PM, Tim Bell  wrote:

>
> Looking at projects like Barbican in the
> https://www.openstack.org/software/releases/liberty/components/barbican,
> they seem to have met some conditions which are currently marked as ‘no’.
>
> What is the mechanism to query these settings (either manual or automatic)
> ?
>
> As an example, there is significant documentation for Barbican at
> http://docs.openstack.org/developer/barbican/ but it is marked as ‘no’ in
> the page for
> https://www.openstack.org/software/releases/liberty/components/barbican  Is
> there an install guide for this project guide (at docs.openstack.org)?
>
> How about adding
>
>- some algorithmic description (i.e. automatic or manual) for the
>selection ? If automatic, the algorithm could be provided so the project
>knows what to do
>- a process description for the manual cases for projects to ask for a
>re-classification
>
> Tim
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>


-- 
Best Regards ,

The G.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [tags] Appealing for change of tags

[Shamail Tahir]

Hi,

We could definitely use more attendance/help on the ops-tag team... The
data in project navigator comes from multiple sources but the install guide
availability[1], packages[2], etc. are derived from ops-tags[3].  The team
meets once a month (3rd Thursday of every month) to review open tags.  I
agree with Tim that we should determine how we can automate the information
collection as much as possible... It's not abandoned but more participation
would be helpful.  We are nearing the point where we should start
populating tags for Mitaka.

[1]
https://github.com/openstack/ops-tags-team/blob/master/liberty/ops-docs-install-guide.json
[2]
https://github.com/openstack/ops-tags-team/blob/master/liberty/ops-packaged.json
[3] https://wiki.openstack.org/wiki/Operations/Tags

Thanks,
Shamail


On Wed, Mar 23, 2016 at 12:12 AM, Hauke Bruno Wollentin <
hauke-bruno.wollen...@innovo-cloud.de> wrote:

> fullack with both of you. The project guide itself is an awesome idea but
> don't know why, it looks a bit abandoned for now. Would be great if someone
> - maybe from the cross project team(?) - could point out how we all could
> make things better (or even who the right person to talk to is ;)).
>
> just my 2 cents,
> cheers,
> hauke
>
>
> *From:* Gal Sagie 
> *Sent:* Mar 23, 2016 11:00 AM
> *To:* Tim Bell
> *Cc:* openstack-operators
> *Subject:* Re: [Openstack-operators] [tags] Appealing for change of tags
>
> The projects themselves don't seems updated, there are other projects in
> the big-tent that are not
> listed there yet.
>
> On Tue, Mar 22, 2016 at 9:16 PM, Tim Bell  wrote:
>
>>
>> Looking at projects like Barbican in the
>> https://www.openstack.org/software/releases/liberty/components/barbican,
>> they seem to have met some conditions which are currently marked as ‘no’.
>>
>> What is the mechanism to query these settings (either manual or
>> automatic) ?
>>
>> As an example, there is significant documentation for Barbican at
>> http://docs.openstack.org/developer/barbican/ but it is marked as ‘no’
>> in the page for
>> https://www.openstack.org/software/releases/liberty/components/barbican  Is
>> there an install guide for this project guide (at docs.openstack.org)?
>>
>> How about adding
>>
>>- some algorithmic description (i.e. automatic or manual) for the
>>selection ? If automatic, the algorithm could be provided so the project
>>knows what to do
>>- a process description for the manual cases for projects to ask for
>>a re-classification
>>
>> Tim
>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
>
> --
> Best Regards ,
>
> The G.
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>


-- 
Thanks,
Shamail Tahir
t: @ShamailXD
tz: Eastern Time
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] create image from paused VM refused with http 409

[Saverio Proto]

I am not a developer but I tried my best !
https://review.openstack.org/#/c/295865/

I applied the patch in my staging system and it fixes the problem :)

Saverio


2016-03-22 13:46 GMT+01:00 Saverio Proto :
> I found the problem. It happens only then the instance has booted from volume.
>
> I think api.py should be patched at the function
> snapshot_volume_backed (line 2250) in a similar way as
> https://review.openstack.org/#/c/116789/
>
> Looks like the bug is also there in master , file nova/compute/api.py line 
> 2296
>
> I will try to submit a patch with Gerrit
>
> Saverio
>
> 2016-03-22 13:33 GMT+01:00 Saverio Proto :
>> Hello there,
>>
>> I used to do this in Juno, and now I upgraded to Kilo and it is not
>> working anymore.
>>
>> macsp:~ proto$ openstack server image create --name test
>> 81da19c6-efbe-4002-b4e8-5ce352ffdd14
>> Cannot 'createImage' instance 81da19c6-efbe-4002-b4e8-5ce352ffdd14
>> while it is in vm_state paused (HTTP 409) (Request-ID:
>> req-0e7f339d-a236-4584-a44c-49daed7558ee)
>>
>> Is this a change of behaviour ?
>>
>> I also found this:
>>
>> https://review.openstack.org/#/c/116789/
>>
>> Should not this work ? my nova-api --version is 2015.1.2
>>
>> Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators