Re: [openstack-dev] [nova] I'm going to expire open bug reports older than 18 months.

[Shoham Peller]

I support Clint's comment, and as an example, only today I was able to
search a bug and to see it was reported 2 years ago and wasn't solved since.
I've commented on the bug saying it happened to me in an up-to-date nova.
I'm talking about a bug which is on your list -
https://bugs.launchpad.net/nova/+bug/1298075

I guess I wouldn't
 been able to do so if the bug was closed.

On Mon, May 30, 2016 at 9:37 PM, Clint Byrum  wrote:

> (Top posting as a general reply to the thread)
>
> Bugs are precious data. As much as it feels like the bug list is full of
> cruft that won't ever get touched, one thing that we might be missing in
> doing this is that the user who encounters the bug and takes the time
> to actually find the bug tracker and report a bug, may be best served
> by finding that somebody else has experienced something similar. If you
> close this bug, that user is now going to be presented with the "I may
> be the first person to report this" flow instead of "yeah I've seen that
> error too!". The former can be a daunting task, but the latter provides
> extra incentive to press forward, since clearly there are others who
> need this, and more data is helpful to triagers and fixers.
>
> I 100% support those who are managing bugs doing whatever they need
> to do to make sure users' issues are being addressed as well as can be
> done with the resources available. However, I would also urge everyone
> to remember that the bug tracker is not only a way for developers to
> manage the bugs, it is also a way for the community of dedicated users
> to interact with the project as a whole.
>
> Excerpts from Markus Zoeller's message of 2016-05-23 13:02:29 +0200:
> > TL;DR: Automatic closing of 185 bug reports which are older than 18
> > months in the week R-13. Skipping specific bug reports is possible. A
> > bug report comment explains the reasons.
> >
> >
> > I'd like to get rid of more clutter in our bug list to make it more
> > comprehensible by a human being. For this, I'm targeting our ~185 bug
> > reports which were reported 18 months ago and still aren't in progress.
> > That's around 37% of open bug reports which aren't in progress. This
> > post is about *how* and *when* I do it. If you have very strong reasons
> > to *not* do it, let me hear them.
> >
> > When
> > 
> > I plan to do it in the week after the non-priority feature freeze.
> > That's week R-13, at the beginning of July. Until this date you can
> > comment on bug reports so they get spared from this cleanup (see below).
> > Beginning from R-13 until R-5 (Newton-3 milestone), we should have
> > enough time to gain some overview of the rest.
> >
> > I also think it makes sense to make this a repeated effort, maybe after
> > each milestone/release or monthly or daily.
> >
> > How
> > ---
> > The bug reports which will be affected are:
> > * in status: [new, confirmed, triaged]
> > * AND without assignee
> > * AND created at: > 18 months
> > A preview of them can be found at [1].
> >
> > You can spare bug reports if you leave a comment there which says
> > one of these (case-sensitive flags):
> > * CONFIRMED FOR: NEWTON
> > * CONFIRMED FOR: MITAKA
> > * CONFIRMED FOR: LIBERTY
> >
> > The expired bug report will have:
> > * status: won't fix
> > * assignee: none
> > * importance: undecided
> > * a new comment which explains *why* this was done
> >
> > The comment the expired bug reports will get:
> > This is an automated cleanup. This bug report got closed because
> > it is older than 18 months and there is no open code change to
> > fix this. After this time it is unlikely that the circumstances
> > which lead to the observed issue can be reproduced.
> > If you can reproduce it, please:
> > * reopen the bug report
> > * AND leave a comment "CONFIRMED FOR: "
> >   Only still supported release names are valid.
> >   valid example: CONFIRMED FOR: LIBERTY
> >   invalid example: CONFIRMED FOR: KILO
> > * AND add the steps to reproduce the issue (if applicable)
> >
> >
> > Let me know if you think this comment gives enough information how to
> > handle this situation.
> >
> >
> > References:
> > [1] http://45.55.105.55:8082/bugs-dashboard.html#tabExpired
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] RPC Communication Errors Might Lead to a Bad State

[Shoham Peller]

An RPC-message TTL mechanism sounds like a good solution.

I've opened a launchpad bug, so we can move the discussion there, and see
if we can think of more ideas to solve this:
https://bugs.launchpad.net/nova/+bug/1571175

Also, please see this previous bug on the same issue:
https://bugs.launchpad.net/nova/+bug/1276214

There, the bug was on a migration request. There, the solution was to catch
the RPC-Timeout exception, since the migrate RPC request is issued using a
"call", and not a "cast", hence waiting for nova-conductor to acknowledge
the message before considering it delivered.
We should either consider moving all RPC operations to 'call', or, if we
want to return from the REST request as fast as possible, let's move the
migrate RPC request to 'cast' as well.

I've added all this information to the launchpad bug.

Thanks,
Shoham
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] RPC Communication Errors Might Lead to a Bad State

[Shoham Peller]

Hi all,

There are some cases that a communication failure between the different
nova services, might cause a bad state in the system.

For example, when "shelving" a VM, nova-api puts the VM's task_state as
"shelving", sends an RPC to nova-compute, which shelves the VM, and resets
it's task_state in DB.
But, if for some reason, nova-compute didn't get the message (i.e. the RPC
service was down, there's a bug in the RPC service, nova-compute was down,
there was a temporary network malfunction), the VM is now stuck as
"shelving", and the user can't perform any operation on the stuck VM.
This example applies to a couple of scenarios in the system that involve
communication between different services.

>From nova-api's point-of-view, all it does is sending a message through
RPC, and neither actually checks that the message was received, nor waits
to get a reply or an acknowledgement from the receiver.

Of course, to solve this, a user can "reset-state" on a VM, and try to run
the action again, but this is error-prone and doesn't scale.

Possible solutions might be:

   - nova-api should receive an acknowledgement from nova-compute. It is
   unclear to me why today it uses a non-reply mechanism - probably to free
   the worker as fast as it can.
   - Change the task_state mechanism to prevent this kind of a stuck state
   to stay in the DB. nova-compute can be the one that writes the task_state
   to the DB, but this is not enough of course, but maybe there's another way?
   - nova-api could start a timer for the action to complete. If the
   shelving operation hasn't completed in X seconds, it will clean it by
   itself and rollback\try-again.

What do you think about the problem and the solutions?

Thanks,
Shoham Peller
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Trailing Slashes on URLs

[Shoham Peller]

I think it should be documented, at least so that tempest can test things
that are actually known and documented behaviors

Thanks,
Shoham

On Wed, Mar 23, 2016 at 1:07 PM, Sean Dague <s...@dague.net> wrote:

> On 03/22/2016 04:44 PM, Shoham Peller wrote:
> > Thank you for your answer Sean. Well understood.
> > However, I think that if not fix this issue, at least we should supply
> > guidelines on the matter.
> >
> > For example tempest's "test_delete_security_group_without_passing_id"
> > test actually checks this behavior by addressing a security group with
> > an "empty id" (/v2.1/{tenant_id}/os-security-groups/), and expects nova
> > to return 404. The comment on the code says:
> > # Negative test:Deletion of a Security Group with out passing ID should
> > Fail
> >
> > If we'll release guidelines on the matter, and the right way to use the
> > API, and that the trailing slashes issue is just a known bug as you say,
> > this test is rendered useless.
> >
> > What do you think?
>
> I think it's known behavior, I'm still not convinced it's a bug. It's
> the way we document that things work. If you feel we need to be more
> explicit in documentation about it, I think that would be appropriate.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Trailing Slashes on URLs

[Shoham Peller]

Thank you for your answer Sean. Well understood.
However, I think that if not fix this issue, at least we should supply
guidelines on the matter.

For example tempest's "test_delete_security_group_without_passing_id" test
actually checks this behavior by addressing a security group with an "empty
id" (/v2.1/{tenant_id}/os-security-groups/), and expects nova to return
404. The comment on the code says:
# Negative test:Deletion of a Security Group with out passing ID should
Fail

If we'll release guidelines on the matter, and the right way to use the
API, and that the trailing slashes issue is just a known bug as you say,
this test is rendered useless.

What do you think?

On Tue, Mar 22, 2016 at 7:37 PM, Sean Dague <s...@dague.net> wrote:
>
> On 03/22/2016 12:34 PM, Shoham Peller wrote:
> > Hi,
> >
> > Nova-api behaves different whether a trailing-slash is given in the URL
> > or not.
> > These 2 requests produce different results:
> >
> >   * GET /v2/{tenant-id}/servers - gets a list of servers
> >   * GET /v2/{tenant-id}/servers/ - gets an info for a server with an
> > "empty id" - produces 404
> >
> > IMHO, a trailing slash in REST requests should be ignored.
> >
> > A few resources about REST and trailing slashes:
> >
http://programmers.stackexchange.com/questions/186959/trailing-slash-in-restful-api
> > https://github.com/nicolaiarocci/eve/issues/118
> >
> > What do you think of this issue?
> > Is this difference in behavior is on purpose?
>
> This is the way python routes works, it's pretty embedded behavior in
> libraries we don't control.
>
> While one could argue they should be equivalent, the docs are pretty
> straight forward about what the correct links are, and we only generate
> working links in our references.
>
> There are many issues that we should address in the API, this one
> probably doesn't crack the top 100.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Trailing Slashes on URLs

[Shoham Peller]

Hi,

Nova-api behaves different whether a trailing-slash is given in the URL or
not.
These 2 requests produce different results:

   - GET /v2/{tenant-id}/servers - gets a list of servers
   - GET /v2/{tenant-id}/servers/ - gets an info for a server with an
   "empty id" - produces 404

IMHO, a trailing slash in REST requests should be ignored.

A few resources about REST and trailing slashes:
http://programmers.stackexchange.com/questions/186959/trailing-slash-in-restful-api
https://github.com/nicolaiarocci/eve/issues/118

What do you think of this issue?
Is this difference in behavior is on purpose?

Thanks,
Shoham
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Network operations on shelve_offload'd servers

[Shoham Peller]

Hi,

A recently merged patch from 2 weeks ago allows to attach\detach volumes to
a shelved_offload server:
https://review.openstack.org/#/c/259528/

Network operations on shelved_offload server is currently not allowed.
There's a bug and a change proposal from a year-and-a-half ago regarding
this issue:
https://bugs.launchpad.net/nova/+bug/1299333
https://review.openstack.org/#/c/87081/

In the change, I see that core contributors Dan Smith and Andrew Laski
commented that such operations shouldn't be allowed on shelved servers.

Do the latest volume operations changes mean that approach has changed? I'd
like to revive this change and to re-propose it.

Thank you,
Shoham Peller
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Updating a volume attachment

[Shoham Peller]

Thank you Andrey. I wasn't aware of that spec.

It indeed implements the same idea of updating the bdm, but it's not enough
in my opinion.
It is about updating the delete_on_termination parameter, which is great,
but the user should also be able to update the other parameters, such as
the disk_bus and device_type.

I've added the same comment on the spec itself.

However, seeing the spec both showed me that there is a relative agreement
about the necessity of that change, and the preferable way to update the
API.
The spec is currently abandoned. I will consult with Alexandre, who wrote
the spec, about restoring the spec and maybe expand it to answer for the
updating for other bdm parameters.

Thanks again,
Shoham

On Fri, Feb 12, 2016 at 4:32 PM, Andrey Pavlov <andrey...@gmail.com> wrote:
>
> Hello Shoham,
>
> We've tried to write and implement similar spec [1].
> And someone have tried to implement it [2].
> You can see comments in it.
>
> [1] - https://review.openstack.org/#/c/234269/
> [2] - https://review.openstack.org/#/c/259518/
>
> On Thu, Feb 11, 2016 at 8:20 PM, Shoham Peller
> <shoham.pel...@stratoscale.com> wrote:
> > Thank you Andrea for your reply.
> >
> > I know this spec and it is indeed a viable solution.
> > However, I think allowing users to update the attachment detail, rather
than
> > detach and re-attach a volume for every change is more robust and more
> > convenient.
> >
> > Also, IMHO it's a better user-experience if users can use a single API
call
> > instead of detach API call, poll for the detachment, re-attach the
volume,
> > and poll again for the attachment if they want to powerup the VM.
> > The bdm DB updating, can happen from nova-api, without IRC'ing a compute
> > node, and thus return only when the request has been completed fully.
> >
> > Don't you agree it's needed, even when detaching a boot volume is
possible?
> >
> > Shoham
> >
> > On Thu, Feb 11, 2016 at 7:04 PM, Andrea Rosa <andrea.r...@hpe.com>
wrote:
> >>
> >> Hi
> >>
> >> On 11/02/16 16:51, Shoham Peller wrote:
> >> > if the volume we want to update is the boot
> >> > volume, detaching to update the bdm, is not even possible.
> >>
> >> You might be interested in the approved spec [1] we have for Mitaka
> >> (ref. detach boot volume).
> >> Unfortunately the spec was not part of the high-priority list and it
> >> didn't get implemented but we are going to propose it again for Newton.
> >>
> >> Regards
> >> --
> >> Andrea Rosa
> >>
> >> [1]
> >>
> >>
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/detach-boot-volume.html
> >>
> >>
__
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
__
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> --
> Kind regards,
> Andrey Pavlov.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Updating a volume attachment

[Shoham Peller]

Hi,

Currently there is no way to update the volume-attachment bdm parameters,
i.e. the bus type or device name, without detaching and re-attaching the
volume, and supplying the new parameters. This is obviously not ideal, and
if the volume we want to update is the boot volume, detaching to update the
bdm, is not even possible.

I want to send a spec proposition to allow updating those attachment
parameters, while the server is shutoff of course.

A question is what do you think is the right API for this request.
My first thought is to expand the "os-volume_attachments" API and to add a
PUT method which will get new bdm parameters. Problem is the POST method
doesn't get all the parameters in a bdm - only a device_name.

The options I can think off are:
1. Add to the POST and to the PUT "os-volume_attachments" methods, in the
volumeAttachment dict all the parameters from the bdm that we want the user
to be able to update - probably just device_type and bus_type (device_name
is already present). They will be optional of course, for
backward-compatibility.
2. Instead of expanding the "os-volume_attachments" API, expand the other
way to attach a volume - through a server "attach" action request. We can
add another action - "attachUpdate", which will get the same parameters as
the volume-attach, including the bdm parameters, and with that the user can
update the bdm.

What do you think about the proposal and about the API dilemma?

Thanks,
Shoham
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Updating a volume attachment

[Shoham Peller]

Thank you Andrea for your reply.

I know this spec and it is indeed a viable solution.
However, I think allowing users to update the attachment detail, rather
than detach and re-attach a volume for every change is more robust and more
convenient.

Also, IMHO it's a better user-experience if users can use a single API call
instead of detach API call, poll for the detachment, re-attach the volume,
and poll again for the attachment if they want to powerup the VM.
The bdm DB updating, can happen from nova-api, without IRC'ing a compute
node, and thus return only when the request has been completed fully.

Don't you agree it's needed, even when detaching a boot volume is possible?

Shoham

On Thu, Feb 11, 2016 at 7:04 PM, Andrea Rosa <andrea.r...@hpe.com> wrote:
>
> Hi
>
> On 11/02/16 16:51, Shoham Peller wrote:
> > if the volume we want to update is the boot
> > volume, detaching to update the bdm, is not even possible.
>
> You might be interested in the approved spec [1] we have for Mitaka
> (ref. detach boot volume).
> Unfortunately the spec was not part of the high-priority list and it
> didn't get implemented but we are going to propose it again for Newton.
>
> Regards
> --
> Andrea Rosa
>
> [1]
>
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/detach-boot-volume.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Feature suggestion - API for creating VM without powering it up

[Shoham Peller]

Thank you for your reply.

This approach has 3 disadvantages that I can think of:

1. An image might have configurations processes that only happen on the
first boot and require user input. This is a probable use-case
2. Not all images support cloud-init
3. This creates an unnecessary load on the compute node.

I believe a create-in-shutoff-state is a more scalable, more robust
approach.

Shoham Peller

On Wed, Jan 27, 2016 at 11:06 AM, Rui Chen <chenrui.m...@gmail.com> wrote:

> Looks like we can use user_data and cloud-init to do this stuff.
>
> Adding the following content into user_data.txt and launch instance like
> this: nova boot --user-data user_data.txt ...,
> the instance will shutdown after boot is finished.
>
> power_state:
>  mode: poweroff
>  message: Bye Bye
>
> You can find more details in cloud-init document[1].
>
> [1]:
> https://github.com/number5/cloud-init/blob/master/doc/examples/cloud-config.txt
>
> 2016-01-22 3:32 GMT+08:00 Fox, Kevin M <kevin@pnnl.gov>:
>
>> The nova instance user spec has a use case.
>> https://review.openstack.org/#/c/93/
>>
>> Thanks,
>> Kevin
>> 
>> From: Matt Riedemann [mrie...@linux.vnet.ibm.com]
>> Sent: Thursday, January 21, 2016 7:32 AM
>> To: openstack-dev@lists.openstack.org
>> Subject: Re: [openstack-dev] [nova] Feature suggestion - API for creating
>> VM without powering it up
>>
>> On 1/20/2016 10:57 AM, Shoham Peller wrote:
>> > Hi,
>> >
>> > I would like to suggest a feature in nova to allow creating a VM,
>> > without powering it up.
>> >
>> > If the user will be able to create a stopped VM, it will allow for
>> > better flexibility and user automation.
>> >
>> > I can personally say such a feature would greatly improve comfortability
>> > of my work with nova - currently we shutdown each vm manually as we're
>> > creating it.
>> > What do you think?
>> >
>> > Regards,
>> > Shoham Peller
>> >
>> >
>> >
>> __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> What is your use case?
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Feature suggestion - API for creating VM without powering it up

[Shoham Peller]

Hi,

I would like to suggest a feature in nova to allow creating a VM, without
powering it up.

If the user will be able to create a stopped VM, it will allow for better
flexibility and user automation.

I can personally say such a feature would greatly improve comfortability of
my work with nova - currently we shutdown each vm manually as we're
creating it.
What do you think?

Regards,
Shoham Peller
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev