[openstack-dev] [CFP] FOSDEM 2019 IaaS and Virt DevRoom

[Kashyap Chamarthy]

Dear OpenStack community, 

FOSDEM 2019 will feature a Virtualization & IaaS DevRoom again.  Here is
the call for proposals.  Please check it out if you would like to submit
a talk.

Regards,
Kashyap

---
We are excited to announce that the call for proposals is now open for
the Virtualization & IaaS devroom at the upcoming FOSDEM 2019, to be
hosted on February 2nd 2019.

This year will mark FOSDEM’s 19th anniversary as one of the
longest-running free and open source software developer events,
attracting thousands of developers and users from all over the world.
FOSDEM will be held once again in Brussels, Belgium, on February 2nd &
3rd, 2019.

This devroom is a collaborative effort, and is organized by dedicated
folks from projects such as OpenStack, Xen Project, oVirt, QEMU, KVM,
and Foreman. We would like to invite all those who are involved in these
fields to submit your proposals by December 1st, 2018.

Important Dates
---

Submission deadline: 1 December 2019
Acceptance notifications: 14 December 2019
Final schedule announcement: 21 December 2019
Devroom: 2nd February 2019

About the Devroom
-

The Virtualization & IaaS devroom will feature session topics such as
open source hypervisors and virtual machine managers such as Xen
Project, KVM, bhyve, and VirtualBox, and Infrastructure-as-a-Service
projects such as KubeVirt, Apache CloudStack, OpenStack, oVirt, QEMU and
OpenNebula.

This devroom will host presentations that focus on topics of shared
interest, such as KVM; libvirt; shared storage; virtualized networking;
cloud security; clustering and high availability; interfacing with
multiple hypervisors; hyperconverged deployments; and scaling across
hundreds or thousands of servers.

Presentations in this devroom will be aimed at developers working on
these platforms who are looking to collaborate and improve shared
infrastructure or solve common problems. We seek topics that encourage
dialog between projects and continued work post-FOSDEM.

Submit Your Proposal


All submissions must be made via the Pentabarf event planning site[1].
If you have not used Pentabarf before, you will need to create an
account. If you submitted proposals for FOSDEM in previous years, you
can use your existing account.

After creating the account, select Create Event to start the submission
process. Make sure to select Virtualization and IaaS devroom from the
Track list. Please fill out all the required fields, and provide a
meaningful abstract and description of your proposed session.

Submission Guidelines
-

We expect more proposals than we can possibly accept, so it is vitally
important that you submit your proposal on or before the deadline. Late
submissions are unlikely to be considered.

All presentation slots are 30 minutes, with 20 minutes planned for
presentations, and 10 minutes for Q

All presentations will be recorded and made available under Creative
Commons licenses. In the Submission notes field, please indicate that
you agree that your presentation will be licensed under the CC-By-SA-4.0
or CC-By-4.0 license and that you agree to have your presentation
recorded.

For example:

"If my presentation is accepted for FOSDEM, I hereby agree to license
all recordings, slides, and other associated materials under the
Creative Commons Attribution Share-Alike 4.0 International License.
Sincerely, ."

In the Submission notes field, please also confirm that if your talk is
accepted, you will be able to attend FOSDEM and deliver your
presentation.  We will not consider proposals from prospective speakers
who are unsure whether they will be able to secure funds for travel and
lodging to attend FOSDEM. (Sadly, we are not able to offer travel
funding for prospective speakers.)
Speaker Mentoring Program

As a part of the rising efforts to grow our communities and encourage a
diverse and inclusive conference ecosystem, we're happy to announce that
we'll be offering mentoring for new speakers. Our mentors can help you
with tasks such as reviewing your abstract, reviewing your presentation
outline or slides, or practicing your talk with you.

You may apply to the mentoring program as a newcomer speaker if you:

Never presented before or
Presented only lightning talks or
Presented full-length talks at small meetups (<50 ppl)

Submission Guidelines
-

Mentored presentations will have 25-minute slots, where 20 minutes will
include the presentation and 5 minutes will be reserved for questions.
The number of newcomer session slots is limited, so we will probably not
be able to accept all applications.

You must submit your talk and abstract to apply for the mentoring
program, our mentors are volunteering their time and will happily
provide feedback but won't write your presentation for you!

If you are experiencing problems with Pentabarf, the proposal submission
interface, or 

[openstack-dev] RFC: Next minimum libvirt / QEMU versions for 'T' release

[Kashyap Chamarthy]

Hey folks,

Before we bump the agreed upon[1] minimum versions for libvirt and QEMU
for 'Stein', we need to do the tedious work of picking the NEXT_MIN_*
versions for the 'T' (which is still in the naming phase) release, which
will come out in the autumn (Sep-Nov) of 2019.

Proposal


Looking at the DistroSupportMatrix[2], it seems like we can pick the
libvirt and QEMU versions supported by the next LTS release of Ubuntu --
18.04; "Bionic", which are:

libvirt: 4.0.0
QEMU:2.11

Debian, Fedora, Ubuntu (Bionic), openSUSE currently already ship the
above versions.  And it seems reasonable to assume that the enterprise
distribtions will also ship the said versions pretty soon; but let's
double-confirm below.

Considerations and open questions
-

(a) KVM for IBM z Systems: John Garbutt pointed out[3] on IRC that:
"IBM announced that KVM for IBM z will be withdrawn, effective March
31, 2018 [...] development will not only continue unaffected, but
the options for users grow, especially with the recent addition of
SuSE to the existing support in Ubuntu."

The message seems to be: "use a regular distribution".  So this is
covered, if we a version based on other distributions.

(b) Oracle Linux: Can you please confirm if you'll be able to
release libvirt and QEMU to 4.0.0 and 2.11, respectively?

(c) SLES: Same question as above.

Assuming Oracle Linux and SLES confirm, please let us know if there are
any objections if we pick NEXT_MIN_* versions for the OpenStack 'T'
release to be libvirt: 4.0.0 and QEMU: 2.11.

* * *

A refresher on libvirt and QEMU release schedules
-

  - There will be at least 12 libvirt releases (_excluding_ maintenance
releases) by Autumn 2019.  A new libvirt release comes out every
month[4].

  - And there will be about 4 releases of QEMU.  A new QEMU release
comes out once every four months.

[1] http://git.openstack.org/cgit/openstack/nova/commit/?h=master=28d337b
-- Pick next minimum libvirt / QEMU versions for "Stein"
[2] https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
[3] http://kvmonz.blogspot.com/2017/03/kvm-for-ibm-z-withdrawal.html
[4] https://libvirt.org/downloads.html#schedule

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [nova] [placement] placement below or beside compute after extraction?

[Kashyap Chamarthy]

On Wed, Aug 22, 2018 at 11:03:43AM -0700, melanie witt wrote:

[...]

[Randomly jumping in on one specific point.]

> Aside from that, it has always been difficult to add folks to
> nova-core because of the large scope and expertise needed to approve
> code across all of Nova.

The complexity of Nova, and the amount of context one needs to keep in
their head will only _keep_ increasing.  Thus, that "difficult to add
folks" becomes a self-perpetuating problem.

And as we know, not every Nova contributor would want to learn the
_whole_ of Nova — so, for the vanishingly small portion of people who
might want to learn "all of Nova", it will be an uphill battle where the
hill is only going to get steeper.  Some people spend all of their time
on specific subsystems of Nova (scheduler, virt drivers, etc); yet
others work on unrelated projects (that don't overlap with OpenStack,
but are "critical dependencies" for Nova and OpenStack) and thus have
limited time for Nova, and so forth.

This reminds me of the highly articulate thread[1] from Dan Berrangé in
2014.  It would be educating to see how we stand today, in relation to
the points raised in that thread, after four years.

[1] 
http://lists.openstack.org/pipermail/openstack-dev/2014-September/044872.html

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] increasing the number of allowed volumes attached per instance > 26

[Kashyap Chamarthy]

On Mon, Jun 11, 2018 at 10:14:33AM -0500, Eric Fried wrote:
> I thought we were leaning toward the option where nova itself doesn't
> impose a limit, but lets the virt driver decide.

Yeah, I agree with that, if we can't arrive at a sensible limit for
Nova, after testing with all drivers that matter (which I doubt will
happen anytime soon).

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] increasing the number of allowed volumes attached per instance > 26

[Kashyap Chamarthy]

On Mon, Jun 11, 2018 at 11:55:29AM +0200, Sahid Orentino Ferdjaoui wrote:
> On Fri, Jun 08, 2018 at 11:35:45AM +0200, Kashyap Chamarthy wrote:
> > On Thu, Jun 07, 2018 at 01:07:48PM -0500, Matt Riedemann wrote:

[...]

> > > The 26 volumes thing is a libvirt driver restriction.
> > 
> > The original limitation of 26 disks was because at that time there was
> > no 'virtio-scsi'.  
> > 
> > (With 'virtio-scsi', each of its controller allows upto 256 targets, and
> > each target can use any LUN (Logical Unit Number) from 0 to 16383
> > (inclusive).  Therefore, the maxium allowable disks on a single
> > 'virtio-scsi' controller is 256 * 16384 == 4194304.)  Source[1].
> 
> Not totally true for Nova. Nova handles one virtio-scsi controller per
> guest and plug all the volumes in one target so in theory that would
> be 16384 LUN (only).

Yeah, I could've been clearer that I was only talking the maximum
allowable disks regardless of how Nova handles it.

> But you made a good point the 26 volumes thing is not a libvirt driver
> restriction. For example the QEMU SCSI native implementation handles
> 256 disks.
> 
> About the virtio-blk limitation I made the same finding but Tsuyoshi
> Nagata shared an interesting point saying that virtio-blk is not longer
> limited by the number of PCI slot available. That in recent kernel and
> QEMU version [0].
> 
> I could join what you are suggesting at the bottom and fix the limit
> to 256 disks.

Right, that's for KVM-based hypervisors.  

Eric Fried on IRC said the other day that for IBM POWER hypervisor they
have tested (not with OpenStack) upto 4000 disks.  But I am yet to see
any more concrete details from POWER hypervisor users on this thread.

If people can't seem to reach an agreement on the limits, we may have to
settle with conditionals:

if kvm|qemu:
return 256
elif POWER:
return 4000
elif:
...

Before that we need concrete data that it is a _reasonble_ limit for
POWER hypervisor (and possibly others).

> [0] 
> https://review.openstack.org/#/c/567472/16/nova/virt/libvirt/blockinfo.py@162

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] increasing the number of allowed volumes attached per instance > 26

[Kashyap Chamarthy]

On Thu, Jun 07, 2018 at 01:07:48PM -0500, Matt Riedemann wrote:
> On 6/7/2018 12:56 PM, melanie witt wrote:
> > Recently, we've received interest about increasing the maximum number of
> > allowed volumes to attach to a single instance > 26. The limit of 26 is
> > because of a historical limitation in libvirt (if I remember correctly)
> > and is no longer limited at the libvirt level in the present day. So,
> > we're looking at providing a way to attach more than 26 volumes to a
> > single instance and we want your feedback.
> 
> The 26 volumes thing is a libvirt driver restriction.

The original limitation of 26 disks was because at that time there was
no 'virtio-scsi'.  

(With 'virtio-scsi', each of its controller allows upto 256 targets, and
each target can use any LUN (Logical Unit Number) from 0 to 16383
(inclusive).  Therefore, the maxium allowable disks on a single
'virtio-scsi' controller is 256 * 16384 == 4194304.)  Source[1].

[...]

> > Some ideas that have been discussed so far include:
> > 
> > A) Selecting a new, higher maximum that still yields reasonable
> > performance on a single compute host (64 or 128, for example). Pros:
> > helps prevent the potential for poor performance on a compute host from
> > attaching too many volumes. Cons: doesn't let anyone opt-in to a higher
> > maximum if their environment can handle it.

Option (A) can still be considered: We can limit it to 256 disks.  Why?

FWIW, I did some digging here:

The upstream libguestfs project after some thorough testing, arrived at
a limit of 256 disks, and suggest the same for Nova.  And if anyone
wants to increase that limit, the proposer should come up with a fully
worked through test plan. :-) (Try doing any meaningful I/O to so many
disks at once, and see how well that works out.)

What more, the libguestfs upstream tests 256 disks, and even _that_
fails sometimes:

https://bugzilla.redhat.com/show_bug.cgi?id=1478201 -- "kernel runs
out of memory with 256 virtio-scsi disks"

The above bug is fixed now in kernel-4.17.0-0.rc3.git1.2. (And also
required a corresponding fix in QEMU[2], which is available from version
v2.11.0 onwards.)

[...]


[1] https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg02823.html
-- virtio-scsi limits
[2] https://git.qemu.org/?p=qemu.git;a=commit;h=5c0919d 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [StarlingX] StarlingX code followup discussions

[Kashyap Chamarthy]

On Tue, May 22, 2018 at 01:54:59PM -0500, Dean Troyer wrote:
> StarlingX (aka STX) was announced this week at the summit, there is a
> PR to create project repos in Gerrit at [0]. STX is basically Wind

From a cursory look at the libvirt fork, there are some questionable
choices.  E.g. the config code (libvirt/src/qemu/qemu.conf) is modified
such that QEMU is launched as 'root'.  That means a bug in QEMU ==
instant host compromise.

All Linux distributions (that matter) configure libvirt to launch QEMU
as a regular user ('qemu').  E.g. from Fedora's libvirt RPM spec file:

libvirt.spec:%define qemu_user  qemu
libvirt.spec:   --with-qemu-user=%{qemu_user} \

* * *

There are multiple other such issues in the forked libvirt code.

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [StarlingX] StarlingX code followup discussions

[Kashyap Chamarthy]

On Tue, May 22, 2018 at 05:41:18PM -0400, Brian Haley wrote:
> On 05/22/2018 04:57 PM, Jay Pipes wrote:

[...]

> > Please don't take this the wrong way, Dean, but you aren't seriously
> > suggesting that anyone outside of Windriver/Intel would ever contribute
> > to these repos are you?
> > 
> > What motivation would anyone outside of Windriver/Intel -- who must make
> > money on this effort otherwise I have no idea why they are doing it --
> > have to commit any code at all to StarlingX?

Yes, same question as Jay here.

What this product-turned-project (i.e. "Downstream First") is implicitly
asking for is the review time of the upstream community, which is
already at a premium -- for a fork.

> I read this the other way - the goal is to get all the forked code from
> StarlingX into upstream repos.  That seems backwards from how this should
> have been done (i.e. upstream first), and I don't see how a project would
> prioritize that over other work.
> 
> > I'm truly wondering why was this even open-sourced to begin with? I'm as
> > big a supporter of open source as anyone, but I'm really struggling to
> > comprehend the business, technical, or marketing decisions behind this
> > action. Please help me understand. What am I missing?
> 
> I'm just as confused.

Equally stupefied here.

> > My personal opinion is that I don't think that any products, derivatives
> > or distributions should be hosted on openstack.org infrastructure.

Yes, it should be unmistakably clear that contributions to "upstream
Nova", for example, means the 'primary' (this qualifier itself is
redundant) upstream Nova.  No slippery slope such as: "OpenStack-hosted
Nova, but not exactly _that_ OpenStack Nova".

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Kashyap Chamarthy]

On Mon, Apr 09, 2018 at 04:24:06PM -0500, Matt Riedemann wrote:
> On 4/9/2018 4:58 AM, Kashyap Chamarthy wrote:
> > Keep in mind that Matt has a tendency to sometimes unfairly
> > over-simplify others views;-).  More seriously, c'mon Matt; I went out
> > of my way to spend time learning about Debian's packaging structure and
> > trying to get the details right by talking to folks on
> > #debian-backports.  And as you may have seen, I marked the patch[*] as
> > "RFC", and repeatedly said that I'm working on an agreeable lowest
> > common denominator.
> 
> Sorry Kashyap, I didn't mean to offend. I was hoping "delicious bugs" would
> have made that obvious but I can see how it's not. You've done a great,
> thorough job on sorting this all out.

No problem at all.  I know your communication style enough to not take
offence :-).  Thanks for the words!

> Since I didn't know what "RFC" meant until googling it today, how about
> dropping that from the patch so I can +2 it?

Sure, I meant to remove it on my last iteration; now dropped it.  (As
you noted on the review, should've used '-Workflow', but I typed "RFC"
out of muscle memory.)

Thanks for the review.

* * *

Aside: On the other patch[+] that actually bumps for "Rocky" and fixes
the resulting unit test fallout, I intend to fix the rest of the failing
tests sometime this week.  Remaining tests to be fixed:

test_live_migration_update_serial_console_xml
test_live_migration_with_valid_target_connect_addr
test_live_migration_raises_exception
test_virtuozzo_min_version_ok
test_min_version_ppc_ok
test_live_migration_update_graphics_xml
test_min_version_s390_ok

[+] https://review.openstack.org/#/c/558783/ -- libvirt: Bump
MIN_{LIBVIRT,QEMU}_VERSION for "Rocky"

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Kashyap Chamarthy]

On Fri, Apr 06, 2018 at 12:12:31PM -0500, Matt Riedemann wrote:
> On 4/6/2018 12:07 PM, Kashyap Chamarthy wrote:
> > FWIW, I'd suggest so, if it's not too much maintenance.  It'll just
> > spare you additional bug reports in that area, and the overall default
> > experience when dealing with CPU models would be relatively much better.
> > (Another way to look at it is, multiple other "conservative" long-term
> > stable distributions also provide libvirt 3.2.0 and QEMU 2.9.0, so that
> > should give you confidence.)
> > 
> > Again, I don't want to push too hard on this.  If that'll be messy from
> > a package maintainance POV for you / Debian maintainers, then we could
> > settle with whatever is in 'Stretch'.
> 
> Keep in mind that Kashyap has a tendency to want the latest and greatest of
> libvirt and qemu at all times for all of those delicious bug fixes. 

Keep in mind that Matt has a tendency to sometimes unfairly
over-simplify others views ;-).  More seriously, c'mon Matt; I went out
of my way to spend time learning about Debian's packaging structure and
trying to get the details right by talking to folks on
#debian-backports.  And as you may have seen, I marked the patch[*] as
"RFC", and repeatedly said that I'm working on an agreeable lowest
common denominator.

> But we also know that new code also brings new not-yet-fixed bugs.

Yep, of course.

> Keep in mind the big picture here, we're talking about bumping from
> minimum required (in Rocky) libvirt 1.3.1 to at least 3.0.0 (in Stein)
> and qemu 2.5.0 to at least 2.8.0, so I think that's already covering
> some good ground. Let's not get greedy. :)

Sure :-) Also if there's a way we can avoid bugs in the default
experience with minimal effort, we should.

Anyway, there we go: changed the patch[*] to what's in Stretch.

[*] https://review.openstack.org/#/c/558171/

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Kashyap Chamarthy]

On Fri, Apr 06, 2018 at 06:07:18PM +0200, Thomas Goirand wrote:
> On 04/06/2018 12:07 PM, Kashyap Chamarthy wrote:

[...]

> > Note: You don't even have to build the versions from 'Buster', which are
> > quite new.  Just the slightly more conservative libvirt 3.2.0 and QEMU
> > 2.9.0 -- only if it's possbile.
> 
> Actually, for *official* backports, it's the policy to always update to
> wwhatever is in testing until testing is frozen. 

I see.  Sure, that's fine, too (as "Queens" UCA also has it).  Whatever
is efficient and least painful from a maintenance POV.

> I could maintain an unofficial backport in stretch-stein.debian.net
> though.
>
> > That said ... I just spent comparing the release notes of libvirt 3.0.0
> > and libvirt 3.2.0[1][2].  By using libvirt 3.2.0 and QEMU 2.9.0, Debian 
> > users
> > will be spared from a lot of critical bugs (see all the list in [3]) in
> > CPU comparision area.
> > 
> > [1] 
> > https://www.redhat.com/archives/libvirt-announce/2017-April/msg0.html
> > -- Release of libvirt-3.2.0
> > [2] 
> > https://www.redhat.com/archives/libvirt-announce/2017-January/msg3.html
> > --  Release of libvirt-3.0.0
> > [3] https://www.redhat.com/archives/libvir-list/2017-February/msg01295.html
> 
> So, because of these bugs, would you already advise Nova users to use
> libvirt 3.2.0 for Queens?

FWIW, I'd suggest so, if it's not too much maintenance.  It'll just
spare you additional bug reports in that area, and the overall default
experience when dealing with CPU models would be relatively much better.
(Another way to look at it is, multiple other "conservative" long-term
stable distributions also provide libvirt 3.2.0 and QEMU 2.9.0, so that
should give you confidence.)

Again, I don't want to push too hard on this.  If that'll be messy from
a package maintainance POV for you / Debian maintainers, then we could
settle with whatever is in 'Stretch'.

Thanks for looking into it.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Kashyap Chamarthy]

On Thu, Apr 05, 2018 at 06:11:26PM -0500, Matt Riedemann wrote:
> On 4/5/2018 3:32 PM, Thomas Goirand wrote:
> > If you don't absolutely need new features from libvirt 3.2.0 and 3.0.0
> > is fine, please choose 3.0.0 as minimum.
> > 
> > If you don't absolutely need new features from qemu 2.9.0 and 2.8.0 is
> > fine, please choose 2.8.0 as minimum.
> > 
> > If you don't absolutely need new features from libguestfs 1.36 and 1.34
> > is fine, please choose 1.34 as minimum.
> 
> New features in the libvirt driver which depend on minimum versions of
> libvirt/qemu/libguestfs (or arch for that matter) are always conditional, so
> I think it's reasonable to go with the lower bound for Debian. We can still
> support the features for the newer versions if you're running a system with
> those versions, but not penalize people with slightly older versions if not.

Yep, we can trivially set the lower bound to versions in 'Stretch'.  The
intention was never to "penalize" distributions w/ older versions.  I
was just checking if Debian 'Stretch' users can be spared from the
myriad of CPU-modelling related issues (see my other reply for
specifics) that are all fixed with 3.2.0 (and QMEU 2.9.0) by default --
without spending inordinate amounts of time and messy backporting
procedures.  Since rest of all the other stable distributions are using
it.

I'll wait a day to hear from Zigo, then I'll just rewrite the patch[*] to
use what's currently in 'Stretch'.

[*] https://review.openstack.org/#/c/558171/

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Kashyap Chamarthy]

On Thu, Apr 05, 2018 at 10:32:13PM +0200, Thomas Goirand wrote:

Hey Zigo, thanks for the detailed response; a couple of comments below.

[...]

> backport of libvirt/QEMU/libguestfs more in details
> ---
> 
> I already attempted the backports from Debian Buster to Stretch.
>
> All of the 3 components (libvirt, qemu & libguestfs) could be built
> without extra dependency, which is a very good thing.
> 
> - libvirt 4.1.0 compiled without issue, though the dh_install phase
> failed with this error:
> 
> dh_install: Cannot find (any matches for) "/usr/lib/*/wireshark/" (tried
> in "." and "debian/tmp")
> dh_install: libvirt-wireshark missing files: /usr/lib/*/wireshark/
> dh_install: missing files, aborting

That seems like a problem in the Debian packaging system, not in
libvirt.  I double-checked with the upstream folks, and the install
rules for Wireshark plugin doesn't have /*/ in there.

> - qemu 2.11 built perfectly with zero change.
> 
> - libguestfs 1.36.13 only needed to have fdisk replaced by util-linux as
> build-depends (fdisk is now a separate package in Buster).

Great.

Note: You don't even have to build the versions from 'Buster', which are
quite new.  Just the slightly more conservative libvirt 3.2.0 and QEMU
2.9.0 -- only if it's possbile.

[...]

> Conclusion:
> ---
> 
> If you don't absolutely need new features from libvirt 3.2.0 and 3.0.0
> is fine, please choose 3.0.0 as minimum.
> 
> If you don't absolutely need new features from qemu 2.9.0 and 2.8.0 is
> fine, please choose 2.8.0 as minimum.
> 
> If you don't absolutely need new features from libguestfs 1.36 and 1.34
> is fine, please choose 1.34 as minimum.
> 
> If you do need these new features, I'll do my best adapt. :)

Sure, can use the 3.0.0 (& QEMU 2.8.0), instead of 3.2.0, as we don't
want to "penalize" (that was never the intention) distros with slightly
older versions.

That said ... I just spent comparing the release notes of libvirt 3.0.0
and libvirt 3.2.0[1][2].  By using libvirt 3.2.0 and QEMU 2.9.0, Debian users
will be spared from a lot of critical bugs (see all the list in [3]) in
CPU comparision area.

[1] https://www.redhat.com/archives/libvirt-announce/2017-April/msg0.html
-- Release of libvirt-3.2.0
[2] https://www.redhat.com/archives/libvirt-announce/2017-January/msg3.html
--  Release of libvirt-3.0.0
[3] https://www.redhat.com/archives/libvir-list/2017-February/msg01295.html


[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Kashyap Chamarthy]

On Sat, Mar 31, 2018 at 04:09:29PM +0200, Kashyap Chamarthy wrote:
> [Meta comment: corrected the email subject: "Solar" --> "Stein"]

Here's a change to get the discussion rolling:

https://review.openstack.org/#/c/558171/ -- [RFC] Pick next minimum
libvirt / QEMU versions for "Stein"

> On Fri, Mar 30, 2018 at 04:26:43PM +0200, Kashyap Chamarthy wrote:

[...]

> > Taking the DistroSupportMatrix into picture, for the sake of discussion,
> > how about the following NEXT_MIN versions for "Solar" release:  
> >
> > 
> > (a) libvirt: 3.2.0 (released on 23-Feb-2017)   
> > 
> > This satisfies most distributions, but will affect Debian "Stretch",
> >
> > as they only have 3.0.0 in the stable branch -- I've checked their
> > repositories[3][4].  Although the latest update for the stable
> > release "Stretch (9.4)" was released only on 10-March-2018, I don't
> > think they increment libvirt and QEMU versions in stable.  Is
> > there another way for "Stretch (9.4)" users to get the relevant
> > versions from elsewhere?

I learn that there's Debian 'stretch-backports'[0], which might provide (but
doesn't yet) a newer stable version.

> > (b) QEMU: 2.9.0 (released on 20-Apr-2017)  
> > 
> > This too satisfies most distributions but will affect Oracle Linux
> > -- which seem to ship QEMU 1.5.3 (released in August 2013) with
> > their "7", from the Wiki.  And will also affect Debian "Stretch" --
> > as it only has 2.8.0
> > 
> > Can folks chime in here?

Answering my own questions about Debian --

From looking at the Debian Archive[1][2], these are the versions for
'Stretch' (the current stable release) and in the upcoming 'Buster'
release:

libvirt | 3.0.0-4+deb9u2 | stretch
libvirt | 4.1.0-2| buster

qemu| 1:2.8+dfsg-6+deb9u3| stretch
qemu| 1:2.11+dfsg-1  | buster

I also talked on #debian-backports IRC channel on OFTC network, where I
asked: 

"What I'm essentially looking for is: "How can 'stretch' users get
libvirt 3.2.0 and QEMU 2.9.0, even if via a different repository.
As they are proposed to be least common denominator versions across
distributions."

And two people said: Then the versions from 'Buster' could be backported
to 'stretch-backports'.  The process for that is to: "ask the maintainer
of those package and Cc to the backports mailing list."

Any takers?

[0] https://packages.debian.org/stretch-backports/
[1] https://qa.debian.org/madison.php?package=libvirt
[2] https://qa.debian.org/madison.php?package=qemu

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Kashyap Chamarthy]

[Meta comment: corrected the email subject: "Solar" --> "Stein"]

On Fri, Mar 30, 2018 at 04:26:43PM +0200, Kashyap Chamarthy wrote:
> The last version bump was in "Pike" release (commit: b980df0,
> 11-Feb-2017), and we didn't do any bump during "Queens".  So it's time
> to increment the versions (which will also makes us get rid of some
> backward compatibility cruft), and pick future versions of libvirt and
> QEMU.  
> 
> As it stands, during the "Pike" release the advertized NEXT_MIN versions  
>  
> were set to: libvirt 1.3.1 and QEMU 2.5.0 -- but they weren't actually
>  
> bumped for the "Queens" release.  So they will now be applied for the 
>  
> "Rocky" release.  (Hmm, but note that libvirt 1.3.1 was released more 
>  
> than 2 years ago[1].)  
> 
> While at it, we should also discuss about what will be the NEXT_MIN   
>  
> libvirt and QEMU versions for the "Solar" release.  To that end, I've 
>  
> spent going through different distributions and updated the   
>  
> DistroSupportMatrix Wiki[2].   
> 
> Taking the DistroSupportMatrix into picture, for the sake of discussion,
> how about the following NEXT_MIN versions for "Solar" release:
>  
> 
> (a) libvirt: 3.2.0 (released on 23-Feb-2017)   
> 
> This satisfies most distributions, but will affect Debian "Stretch",  
>  
> as they only have 3.0.0 in the stable branch -- I've checked their
> repositories[3][4].  Although the latest update for the stable
> release "Stretch (9.4)" was released only on 10-March-2018, I don't
> think they increment libvirt and QEMU versions in stable.  Is
> there another way for "Stretch (9.4)" users to get the relevant
> versions from elsewhere?   
> 
> (b) QEMU: 2.9.0 (released on 20-Apr-2017)  
> 
> This too satisfies most distributions but will affect Oracle Linux
> -- which seem to ship QEMU 1.5.3 (released in August 2013) with
> their "7", from the Wiki.  And will also affect Debian "Stretch" --
> as it only has 2.8.0
> 
> Can folks chime in here?
> 
> [1] 
> https://www.redhat.com/archives/libvirt-announce/2016-January/msg2.html
> [2] https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
> [3] https://packages.qa.debian.org/libv/libvirt.html
> [4] https://packages.qa.debian.org/libv/libvirt.html
> 
> -- 
> /kashyap

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] RFC: Next minimum libvirt / QEMU versions for "Solar" release

[Kashyap Chamarthy]

On Sat, Mar 31, 2018 at 03:17:52PM +0200, Kashyap Chamarthy wrote:
> On Fri, Mar 30, 2018 at 09:49:17AM -0500, Sean McGinnis wrote:

[...]

> > > Taking the DistroSupportMatrix into picture, for the sake of discussion,
> > > how about the following NEXT_MIN versions for "Solar" release:
> > >  
> > > 
> > Correction - for the "Stein" release. :)
> 
> Darn, I should've triple-checked before I assumed it is to be "Solar".
> If "Stein" is confirmed; I'll re-send this email with the correct
> release name for clarity.

It actually is:

http://lists.openstack.org/pipermail/openstack-dev/2018-March/128899.html
-- All Hail our Newest Release Name - OpenStack Stein

(That email went into 'openstack-operators' maildir for me; my filtering
fault.)

I won't start another thread;, will just leave this existing thread
intact, as people will read it as: "whatever name the 'S' release ends
up with" (as 'fungi' put it on IRC).

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] RFC: Next minimum libvirt / QEMU versions for "Solar" release

[Kashyap Chamarthy]

On Fri, Mar 30, 2018 at 09:49:17AM -0500, Sean McGinnis wrote:
> > While at it, we should also discuss about what will be the NEXT_MIN 
> >
> > libvirt and QEMU versions for the "Solar" release.  To that end, I've   
> >
> > spent going through different distributions and updated the 
> >
> > DistroSupportMatrix Wiki[2].   
> > 
> > Taking the DistroSupportMatrix into picture, for the sake of discussion,
> > how about the following NEXT_MIN versions for "Solar" release:  
> >
> > 
> Correction - for the "Stein" release. :)

Darn, I should've triple-checked before I assumed it is to be "Solar".
If "Stein" is confirmed; I'll re-send this email with the correct
release name for clarity.

Thanks for correcting.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Solar" release

[Kashyap Chamarthy]

The last version bump was in "Pike" release (commit: b980df0,
11-Feb-2017), and we didn't do any bump during "Queens".  So it's time
to increment the versions (which will also makes us get rid of some
backward compatibility cruft), and pick future versions of libvirt and
QEMU.  

As it stands, during the "Pike" release the advertized NEXT_MIN versions
   
were set to: libvirt 1.3.1 and QEMU 2.5.0 -- but they weren't actually  
   
bumped for the "Queens" release.  So they will now be applied for the   
   
"Rocky" release.  (Hmm, but note that libvirt 1.3.1 was released more   
   
than 2 years ago[1].)  

While at it, we should also discuss about what will be the NEXT_MIN 
   
libvirt and QEMU versions for the "Solar" release.  To that end, I've   
   
spent going through different distributions and updated the 
   
DistroSupportMatrix Wiki[2].   

Taking the DistroSupportMatrix into picture, for the sake of discussion,
how about the following NEXT_MIN versions for "Solar" release:  
   

(a) libvirt: 3.2.0 (released on 23-Feb-2017)   

This satisfies most distributions, but will affect Debian "Stretch",
   
as they only have 3.0.0 in the stable branch -- I've checked their
repositories[3][4].  Although the latest update for the stable
release "Stretch (9.4)" was released only on 10-March-2018, I don't
think they increment libvirt and QEMU versions in stable.  Is
there another way for "Stretch (9.4)" users to get the relevant
versions from elsewhere?   

(b) QEMU: 2.9.0 (released on 20-Apr-2017)  

This too satisfies most distributions but will affect Oracle Linux
-- which seem to ship QEMU 1.5.3 (released in August 2013) with
their "7", from the Wiki.  And will also affect Debian "Stretch" --
as it only has 2.8.0

Can folks chime in here?

[1] https://www.redhat.com/archives/libvirt-announce/2016-January/msg2.html
[2] https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
[3] https://packages.qa.debian.org/libv/libvirt.html
[4] https://packages.qa.debian.org/libv/libvirt.html

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Any reason why not have 'choices' parameter for ListOpt()?

[Kashyap Chamarthy]

On Mon, Mar 26, 2018 at 12:28:02PM -0700, melanie witt wrote:
> On Mon, 26 Mar 2018 14:12:52 -0500, Matt Riedemann wrote:
> > On 3/26/2018 6:24 AM, ChangBo Guo wrote:
> > > What's your use case for ListOpt, just make sure the value(a list) is
> > > part of  'choices' ?   Maybe we need another parameter to distinguish
> > 
> > It came up because of this change in nova:
> > 
> > https://review.openstack.org/#/c/534384/
> > 
> > We want to backport that as a bug fix which is a mitigation for
> > performance degradation issues with QEMU patches for Spectre and Meltdown.
> > 
> > However, in the backport we wanted to restrict the ListOpt to a single
> > value, "pcid". The idea is to restrict the new option to a single value
> > in stable branches.
> > 
> > Then in master, we could remove the 'choices' kwarg so operators can
> > define their list as they wish.
> > 
> > If we were do implement this generically in ListOpt, I suppose 'choices'
> > would mean that the specified list must be a subset of the defined
> > choices list. So in the backport patch, we'd just have choices=[None,
> > 'pcid'] and you can either specify None or 'pcied' for that option
> > (default to None).
> > 
> > Right now the code that's relying on this option just has a hard-coded
> > check for the value which is OK.
> 
> I'm not sure if this helps, but we do already have some example of a ListOpt
> with 'choices' for the VNC auth_schemes:
> 
> https://github.com/openstack/nova/blob/cd15c3d/nova/conf/vnc.py#L229
> 
> Could we do something similar for the backport of the CPU flags patch?

Ah, interesting pointer.  It seems to work locally, and I updated the
patch with it[*]:

[...]
cfg.ListOpt(
'cpu_model_extra_flags',
item_type=types.String(
choices=['pcid']
),
default=[],
help=""" ... """
[...]

Thanks, Melanie.

[*] https://review.openstack.org/#/c/534384/

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] Any reason why not have 'choices' parameter for ListOpt()?

[Kashyap Chamarthy]

Hi there,

I was looking at oslo_config/cfg.py[*], and the StrOpt() class has
'choices' parameter, to allow a sequence of valid values / tuples of
valid values for descriptions.

However, I don't see the same 'choices' parameter for the ListOpt()
class.  Out of curiosity, is there a reason to not add it for ListOpt()
too?

[*] 
https://git.openstack.org/cgit/openstack/oslo.config/tree/oslo_config/cfg.py#n1271

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] PTL Election Season

[Kashyap Chamarthy]

On Mon, Jan 22, 2018 at 05:09:31PM -0600, Matt Riedemann wrote:

[...]

> To anyone that cares, I don't plan on running for Nova PTL again for the
> Rocky release. Queens was my fourth tour and it's definitely time for
> someone else to get the opportunity to lead here. I don't plan on going
> anywhere and I'll be here to help with any transition needed assuming
> someone else (or a couple of people hopefully) will run in the election.
> It's been a great experience and I thank everyone that has had to put up
> with me and my obsessive paperwork and process disorder in the meantime.

Hey Matt,

Thanks (understatement!) for the all the brilliant work (and also for
all the thankless tasks).  I deeply appreciate how effectively you
handle communication in public and the energy that you bring to the
project.  It is a great joy interacting and working with you.

I continue to be amazed at how you can stay on top of almost everything
(at least give the illusion of it -- it reminds me of the "Sheperd
Tone"[*]), _while_ getting things done.

Glad to hear you're not going anywhere.  

To be continued.

[*] https://en.wikipedia.org/wiki/Shepard_tone
https://www.youtube.com/watch?v=LVWTQcZbLgY

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Kashyap Chamarthy]

On Sat, Dec 16, 2017 at 08:16:13AM -0600, Matt Riedemann wrote:
> On 12/16/2017 5:51 AM, Kashyap Chamarthy wrote:

Hi Matt,

First, thanks for the detailed reply with specific examples.

[...]

> > But one pontential question that comes to mind from that old thread is
> > how is this new proposal of 'one coordinated release per year' going to
> > address the following:
> > 
> >  If your spec didn't make into this year's coordiated release, then
> >  does it have to 'sit out' for one whole year?
> > 
> > The shorter release cycle argument addresses that by not coupling specs
> > to a single release -- where specs can be submitted regardless of a
> > release in question.  Meanwhile, code development of the feature can
> > happen over several cycles.  And periodically re-evaluate it in light of
> > new evidence and design discussions.
> > 
> > As it stands[2], this problem seems mitgated by moving specs across
> > releases and re-approving them (insofar as they make sense).  Maybe in
> > current times, the above specs issue is 'non-problem'.  Happy to be
> > corrected.
> 
> Each project would have to decide what works for them, which might mean one
> or two intermediate releases within the 1 year coordinated release where we
> have a freeze period for new features, and then open up again for new spec
> reviews after the intermediate release to allow new content in again. If no
> one is picking up the intermediate release, then it's just self-imposed to
> force us to (1) work toward a deadline and not let things drag on for a year
> and (2) allow in new specs more than once a year so the above problem
> doesn't happen, or is at least mitigated.

Okay, the above approach of per-project based decision to do
intermediate releases sounds reasonable.

> There are also comments elsewhere in this thread about extending the
> stability period, but again, that's per-project at this point. The proposal
> says nothing about a coordinated stability period.

Noted.  (Haven't caught up with all the responses yet.)

> In the last few cycles we've had I think 2 weeks between the global feature
> freeze and RC1, which isn't much time for stability and flushing out last
> minute bugs.
> 
> If we tried to incorporate both multiple freezes and stability periods, we
> might have something like:
> 
> * 3 months of new dev (maybe freeze specs after a month?)
> * 1 month of no new feature work, only bugs, docs, testing
> * intermediate release
> * 
> 
> That would give you a shorter dev cycle than what we do today (at least in
> nova), and do it three times in a year.
>
> The only contributor problem this solves is people have more opportunities
> to get their spec/new feature considered, at least more than once in a year.
> It does not, however, automatically make those specs/new features a priority
> to get review and help from the core team to shepherd them through the
> pipeline.

Yes, that's a good point.  'Consideration' is one thing, getting the
spec / feature to its logical conclusion is something else.

> There was some discussion about the priority issue during the TC office
> hours a few days ago, and Dan Smith has said it elsewhere in this thread.
> The chances of getting work done is proportional to the shared understanding
> and value of the thing being proposed, including risk/size/complexity etc.

Agreed, there are several complex variables here, and not everything can
be so cut-and-dried.

> Here are two concrete examples from part time contributors for nova:
> 
> 1. 
> https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/rebuild-keypair-reset.html
> 
> 2. 
> https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/scaleio-ephemeral-storage-backend.html
> 
 
[...] # Snip description of the above two examples.

> How do we solve that problem? Get someone from Dell/EMC to be full time
> active in Nova for a year or more and build the trust of the core team to
> want to review this change, and trust that the contributors will be around
> to maintain it after it's merged? That's not a realistic solution.

/me nods; I see your point.  And I agree, the above hypothetical is not
realistic.

> We could dig up the slots/runways idea where we essentially do Kanban and
> this gets queued up and the core team must get it in eventually to start new
> work, regardless of priority.

While I remember the terms "slots/runways" but forgot what the idea
entails (I'll look it up).

> I don't have an answer. There are options. The 1 year release idea doesn't
> magically solve this problem. And arguably doing 3 intermediate releases per
> year, if we did that to solve the "missed the 1 year boat" issue, would a

Re: [openstack-dev] [all] Switching to longer development cycles

[Kashyap Chamarthy]

On Thu, Dec 14, 2017 at 12:02:13PM +0100, Kashyap Chamarthy wrote:
> On Wed, Dec 13, 2017 at 05:17:26PM +0100, Thierry Carrez wrote:

[...]

> > What it means:
> > 
> > - We'd only do one *coordinated* release of the OpenStack components
> > per year, and maintain one stable branch per year

Thinking a bit more, the above reminds of this[*] by DanPB a couple of
years ago, where Dan's 'Modest Proposal' was: "switch to a development
cycle that is exactly 2 months".

But one pontential question that comes to mind from that old thread is
how is this new proposal of 'one coordinated release per year' going to
address the following:

If your spec didn't make into this year's coordiated release, then
does it have to 'sit out' for one whole year? 

The shorter release cycle argument addresses that by not coupling specs
to a single release -- where specs can be submitted regardless of a
release in question.  Meanwhile, code development of the feature can
happen over several cycles.  And periodically re-evaluate it in light of
new evidence and design discussions.

As it stands[2], this problem seems mitgated by moving specs across
releases and re-approving them (insofar as they make sense).  Maybe in
current times, the above specs issue is 'non-problem'.  Happy to be
corrected.


[1] http://lists.openstack.org/pipermail/openstack-dev/2015-February/057614.html
-- Re-evaluating the suitability of the 6 month release cycle

[2] https://specs.openstack.org/openstack/nova-specs/readme.html

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Kashyap Chamarthy]

On Fri, Dec 15, 2017 at 10:11:24AM +0800, Zhenyu Zheng wrote:
> On Thu, Dec 14, 2017 at 7:02 PM, Kashyap Chamarthy <kcham...@redhat.com>
> wrote:

[...]

> > For a relatively mature (~7 years; and ~5 years if we count from the
> > time governance changed to OpenStack Foudation) project, one official
> > contributor gathering per year sounds fine.  And for those that need
> > more face time, people would continue to organize informal meetups.  But
> > admittedly, this shifts the logistics apsect onto contributors -- that's
> > probably okay, many other contributor communities self-organize meetups.

[...]

> The contributors in APAC are growing and wish to be more involved in
> OpenStack, it will be really hard for us to join informal meetups( VISA
> Invitation letters, company support etc.)
> So I really hope the current official technical gathering remains, so that
> we can be more involved with the community.

Hi, we both agree.  I empathize with the Visa troubles.  I actually said
that the main official contributor gathering (PTG) should remain.
Didn't mean to imply that the informal meetups MUST be held; it's
only optional.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Kashyap Chamarthy]

On Wed, Dec 13, 2017 at 05:17:26PM +0100, Thierry Carrez wrote:
> Hi everyone,

Hey Thierry,

> Over the past year, it has become pretty obvious to me that our
> self-imposed rhythm no longer matches our natural pace. It feels like we
> are always running elections, feature freeze is always just around the
> corner, we lose too much time to events, and generally the impression
> that there is less time to get things done. Milestones in the
> development cycles are mostly useless now as they fly past us too fast.
> A lot of other people reported that same feeling.

Thanks for articulating it out loud.  FWIW, I share that same feeling to
an extent, especially as one of those people involved in mulitple
communities (the lower-layer components that OpenStack depends on) and
simply don't have the necessary bandwidth to keep up.

[...]

> In another thread, John Dickinson suggested that we move to one-year
> development cycles, and I've been thinking a lot about it. I now think
> it is actually the right way to reconcile our self-imposed rhythm with
> the current pace of development, and I would like us to consider
> switching to year-long development cycles for coordinated releases as
> soon as possible.
> 
> What it means:
> 
> - We'd only do one *coordinated* release of the OpenStack components per
> year, and maintain one stable branch per year
> - We'd elect PTLs for one year instead of every 6 months
> - We'd only have one set of community goals per year
> - We'd have only one PTG with all teams each year

The above sounds reasonable to me.  

And it should (and will) reduce a lot of breathlessness and 'keeping the
pedal glued to the metal' feeling that comes from being unable to keep
up with things.  Not to mention, the OpenStack Foundation welcomes a bit
of breathing room from having to organize and coordinate so many events.

For a relatively mature (~7 years; and ~5 years if we count from the
time governance changed to OpenStack Foudation) project, one official
contributor gathering per year sounds fine.  And for those that need
more face time, people would continue to organize informal meetups.  But
admittedly, this shifts the logistics apsect onto contributors -- that's
probably okay, many other contributor communities self-organize meetups.

[...]

> When ?

[...]

> So the year-long cycles would ideally start at the beginning of the
> year, when we would organize the yearly PTG. That said, I'm not sure we
> can really afford to keep the current rhythm for one more year before
> switching. That is why I'd like us to consider taking the plunge and
> just doing it for *Rocky*, and have a single PTG in 2018 (in Dublin).

Jumping right in sounds good.  Getting started immediately will also get
us to real world 'data' sooner, rather than dilly-dallying around.

[...]

> So... What do you think ?

I welcome this proposal.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] FOSDEM 2018: Call For Proposals: Virtualization & IaaS DevRoom

[Kashyap Chamarthy]

On Mon, Nov 06, 2017 at 01:37:10PM +0100, Kashyap Chamarthy wrote:
> I'm delighted to announce that the call for proposals is now open for
> the Virtualization & IaaS devroom at the upcoming FOSDEM 2018, to be
> hosted on February 3 and 4, 2018.
> 
> This year will mark FOSDEM’s 18th anniversary as one of the longest-running
> free and open source software developer events, attracting thousands of
> developers and users from all over the world. FOSDEM will be held once
> again in Brussels, Belgium, on February 3 & 4, 2018.
> 
> This devroom is a collaborative effort, and is organized by dedicated
> folks from projects such as OpenStack, Xen Project, oVirt, QEMU, KVM,
> libvirt, and Foreman. We would like to invite all those who are involved
> in these fields to submit your proposals by December 1st, 2017.
> 
> ---
> Important Dates
> ---
> Submission deadline: 01 December 2017

A gentle reminder -- submission deadline is just about a week's time
from now.

> Acceptance notifications: 14 December 2017
> Final schedule announcement: 21 December 2017
> Devroom: 03 and 04 February 2018 (two days- different rooms)

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] FOSDEM 2018: Call For Proposals: Virtualization & IaaS DevRoom

[Kashyap Chamarthy]

I'm delighted to announce that the call for proposals is now open for
the Virtualization & IaaS devroom at the upcoming FOSDEM 2018, to be
hosted on February 3 and 4, 2018.

This year will mark FOSDEM’s 18th anniversary as one of the longest-running
free and open source software developer events, attracting thousands of
developers and users from all over the world. FOSDEM will be held once
again in Brussels, Belgium, on February 3 & 4, 2018.

This devroom is a collaborative effort, and is organized by dedicated
folks from projects such as OpenStack, Xen Project, oVirt, QEMU, KVM,
libvirt, and Foreman. We would like to invite all those who are involved
in these fields to submit your proposals by December 1st, 2017.

---
Important Dates
---
Submission deadline: 01 December 2017
Acceptance notifications: 14 December 2017
Final schedule announcement: 21 December 2017
Devroom: 03 and 04 February 2018 (two days- different rooms)

-
About the Devroom
-

The Virtualization & IaaS devroom will feature session topics such as open
source hypervisors and virtual machine managers such as Xen Project, KVM,
bhyve, and VirtualBox, and Infrastructure-as-a-Service projects such as
Apache CloudStack, OpenStack, oVirt, QEMU, OpenNebula, and Ganeti.

This devroom will host presentations that focus on topics of shared
interest, such as KVM; libvirt; shared storage; virtualized networking;
cloud security; clustering and high availability; interfacing with multiple
hypervisors; hyperconverged deployments; and scaling across hundreds or
thousands of servers.

Presentations in this devroom will be aimed at developers working on these
platforms who are looking to collaborate and improve shared infrastructure
or solve common problems. We seek topics that encourage dialog between
projects and continued work post-FOSDEM.


Submit Your Proposal


All submissions must be made via the Pentabarf event planning site[1]. If
you have not used Pentabarf before, you will need to create an account. If
you submitted proposals for FOSDEM in previous years, you can use your
existing account.

After creating the account, select Create Event to start the submission
process. Make sure to select Virtualization and IaaS devroom from the Track
list. Please fill out all the required fields, and provide a meaningful
abstract and description of your proposed session.

-
Submission Guidelines
-

We expect more proposals than we can possibly accept, so it is vitally
important that you submit your proposal on or before the deadline. Late
submissions are unlikely to be considered.

All presentation slots are 45 minutes, with 35 minutes planned for
presentations, and 10 minutes for Q

All presentations will be recorded and made available under Creative
Commons licenses. In the Submission notes field, please indicate that you
agree that your presentation will be licensed under the CC-By-SA-4.0 or
CC-By-4.0 license and that you agree to have your presentation recorded.
For example:

"If my presentation is accepted for FOSDEM, I hereby agree to license all
recordings, slides, and other associated materials under the Creative
Commons Attribution Share-Alike 4.0 International License. Sincerely,
."

In the Submission notes field, please also confirm that if your talk is
accepted, you will be able to attend FOSDEM and deliver your presentation.
We will not consider proposals from prospective speakers who are unsure
whether they will be able to secure funds for travel and lodging to attend
FOSDEM. (Sadly, we are not able to offer travel funding for prospective
speakers.)

-
Speaker Mentoring Program
-

As a part of the rising efforts to grow our communities and encourage a
diverse and inclusive conference ecosystem, we're happy to announce that
we'll be offering mentoring for new speakers. Our mentors can help you with
tasks such as reviewing your abstract, reviewing your presentation outline
or slides, or practicing your talk with you.

You may apply to the mentoring program as a newcomer speaker if you:

Never presented before or
Presented only lightning talks or
Presented full-length talks at small meetups (<50 ppl)

-
Submission Guidelines
-

Mentored presentations will have 25-minute slots, where 20 minutes will
include the presentation and 5 minutes will be reserved for questions.

The number of newcomer session slots is limited, so we will probably not be
able to accept all applications.

You must submit your talk and abstract to apply for the mentoring program,
our mentors are volunteering their time and will happily provide feedback
but won't write your presentation for you!

If you are experiencing problems with Pentabarf, the proposal submission
interface, or have other questions, you can email our devroom mailing
list[2] and we will try 

[openstack-dev] Call For Proposals: KVM Forum 2017 [Submission deadline: 15-JUN-2017]

[Kashyap Chamarthy]

KVM Forum 2017: Call For Participation
October 25-27, 2017 - Hilton Prague - Prague, Czech Republic

(All submissions must be received before midnight June 15, 2017)
=


KVM Forum is an annual event that presents a rare opportunity
for developers and users to meet, discuss the state of Linux   
virtualization technology, and plan for the challenges ahead.
We invite you to lead part of the discussion by submitting a speaking
proposal for KVM Forum 2017.

At this highly technical conference, developers driving innovation
in the KVM virtualization stack (Linux, KVM, QEMU, libvirt) can
meet users who depend on KVM as part of their offerings, or to
power their data centers and clouds.

KVM Forum will include sessions on the state of the KVM
virtualization stack, planning for the future, and many
opportunities for attendees to collaborate. As we celebrate ten years
of KVM development in the Linux kernel, KVM continues to be a
critical part of the FOSS cloud infrastructure.

This year, KVM Forum is joining Open Source Summit in Prague,
Czech Republic. Selected talks from KVM Forum will be presented on
Wednesday October 25 to the full audience of the Open Source Summit.
Also, attendees of KVM Forum will have access to all of the talks from
Open Source Summit on Wednesday.


===
IMPORTANT DATES
===
Submission deadline: June 15, 2017
Notification: August 10, 2017
Schedule announced: August 17, 2017
Event dates: October 25-27, 2017

http://events.linuxfoundation.org/cfp

Suggested topics:
* Scaling, latency optimizations, performance tuning, real-time guests
* Hardening and security
* New features
* Testing

KVM and the Linux kernel:
* Nested virtualization
* Resource management (CPU, I/O, memory) and scheduling
* VFIO: IOMMU, SR-IOV, virtual GPU, etc.
* Networking: Open vSwitch, XDP, etc.
* virtio and vhost
* Architecture ports and new processor features

QEMU:
* Management interfaces: QOM and QMP
* New devices, new boards, new architectures
* Graphics, desktop virtualization and virtual GPU
* New storage features
* High availability, live migration and fault tolerance
* Emulation and TCG
* Firmware: ACPI, UEFI, coreboot, U-Boot, etc.

Management and infrastructure
* Managing KVM: Libvirt, OpenStack, oVirt, etc.
* Storage: Ceph, Gluster, SPDK, etc.
* Network Function Virtualization: DPDK, OPNFV, OVN, etc.
* Provisioning


===
SUBMITTING YOUR PROPOSAL
===
Abstracts due: June 15, 2017

Please submit a short abstract (~150 words) describing your presentation
proposal. Slots vary in length up to 45 minutes. Also include the proposal
type -- one of:
- technical talk
- end-user talk

Submit your proposal here:
http://events.linuxfoundation.org/cfp
Please only use the categories "presentation" and "panel discussion"

You will receive a notification whether or not your presentation proposal
was accepted by August 10, 2017.

Speakers will receive a complimentary pass for the event. In the instance
that case your submission has multiple presenters, only the primary speaker for 
a
proposal will receive a complimentary event pass. For panel discussions, all
panelists will receive a complimentary event pass.

TECHNICAL TALKS

A good technical talk should not just report on what has happened over
the last year; it should present a concrete problem and how it impacts
the user and/or developer community. Whenever applicable, focus on
work that needs to be done, difficulties that haven't yet been solved,
and on decisions that other developers should be aware of. Summarizing
recent developments is okay but it should not be more than a small
portion of the overall talk.

END-USER TALKS

One of the big challenges as developers is to know what, where and how
people actually use our software. We will reserve a few slots for end
users talking about their deployment challenges and achievements.

If you are using KVM in production you are encouraged submit a speaking
proposal. Simply mark it as an end-user talk. As an end user, this is a
unique opportunity to get your input to developers.

HANDS-ON / BOF SESSIONS

We will reserve some time for people to get together and discuss
strategic decisions as well as other topics that are best solved within
smaller groups.

These sessions will be announced during the event. If you are interested
in organizing such a session, please add it to the list at

  http://www.linux-kvm.org/page/KVM_Forum_2017_BOF

Let people you think who might be interested know about your BOF, and encourage
them to add their names to the wiki page as well. Please try to
add your ideas to the list before KVM Forum starts.


PANEL DISCUSSIONS

If you are proposing a panel discussion, please make sure that you list
all of your potential panelists in your the abstract. We will request full
biographies if a panel is accepted.


===
HOTEL / 

Re: [openstack-dev] [nova] Next minimum libvirt version

[Kashyap Chamarthy]

On Thu, Feb 09, 2017 at 05:29:22PM -0600, Matt Riedemann wrote:
> Since danpb hasn't been around I've sort of forgotten about this, but we
> should talk about bumping the minimum required libvirt version in nova.
> 
> Currently it's 1.2.1 and the next was set to 1.2.9.
> 
> On master we're gating on ubuntu 14.04 which has libvirt 1.3.1 (14.04 had
> 1.2.2).
> 
> If we move to require 1.2.9 that effectively kills 14.04 support for
> devstack + libvirt on master, which is probably OK.
> 
> There is also the distro support wiki [1] which hasn't been updated in
> awhile.
> 
> I'm wondering if 1.2.9 is a safe move for the next required minimum version
> and if so, does anyone have ideas on the next required version after that?

1.2.9 was release on OCT 2014.

And there have been 26 releases from 1.2.9 until now:

1.2.{10,11,12,13,14,15,16,17,18,19,20,21}
1.3.{0,1,2,3,4,5}
2.0.0 [01 JUL 2016]
2.1.0 [02 AUG 2016]
2.2.0 [02 SEP 2016]
2.3.0 [04 OCT 2016]
2.4.0 [01 NOV 2016]
2.5.0 [04 DEC 2016]
3.0.0 [17 JAN 2017]
3.1.0 [Unreleased, as of 10-FEB-2017]

IIUC, going by how[X] Dan settled on 1.1.1, as minimum version for
Newton, it seems like have to mine through the current releases,
that'll: 

  - provide unconditional support for specific features we need; 
  - removes need for maintaining backward compatibility code

[X] 
http://lists.openstack.org/pipermail/openstack-operators/2015-October/008400.html

---

Good news for future: Upstream libvirt recently started an effort to
make release notes more consumable, by writing more 'structured' release
notes, and categorizing the work into: new Features; improvements; and
bug fixes.

You can see the result by comparing this page:

http://libvirt.org/news.html

With the older releases (where it's mostly information from Git
commits):

http://libvirt.org/news-2016.html

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] FOSDEM 2017: Call For Proposals -- Virtualization & IaaS DevRoom

[Kashyap Chamarthy]

===

The call for proposals is now open for the Virtualization & IaaS devroom
at the upcoming FOSDEM 2017, to be hosted on February 4, 2017.

This year will mark FOSDEM’s 17th anniversary as one of the
longest-running free and open source software developer events,
attracting thousands of developers and users from all over the world.
FOSDEM will be held once again in Brussels, Belgium, on February 4 & 5,
2017.

---
Important Dates
---

*Submission deadline*: *18 November 2016*
Acceptance notifications: 04 December 2016
Final schedule announcement: 11 December 2016
Devroom: 04 February 2017 (one day)

-
About the Devroom
-

The Virtualization & IaaS devroom will feature session topics such as
open source hypervisors and virtual machine managers such as Xen
Project, KVM, QEMU, bhyve, and VirtualBox, and
Infrastructure-as-a-Service projects such as Apache CloudStack,
OpenStack, oVirt, OpenNebula, and Ganeti.

This devroom will host presentations that focus on topics of shared
interest, such as KVM; libvirt; shared storage; virtualized networking;
cloud security; clustering and high availability; interfacing with
multiple hypervisors; hyperconverged deployments; and scaling across
hundreds or thousands of servers.

Presentations in this devroom will be aimed at developers working on
these platforms who are looking to collaborate and improve shared
infrastructure or solve common problems. We seek topics that encourage
dialog between projects and continued work post-FOSDEM.


Submit Your Proposal


All submissions must be made via the Pentabarf event planning site.

https://penta.fosdem.org/submission/FOSDEM17

If you have not used Pentabarf before, you will need to create an
account.  If you submitted proposals for FOSDEM in previous years, you
can use your existing account.

After creating the account, select Create Event to start the submission
process. Make sure to select Virtualisation and IaaS devroom from the
Track list. Please fill out all the required fields, and provide a
meaningful abstract and description of your proposed session.

-
Submission Guidelines
-

- We expect more proposals than we can possibly accept, so it is vitally
  important that you submit your proposal on or before the deadline.
  Late submissions are unlikely to be considered.

- All presentation slots are 45 minutes, with 35 minutes planned for
  presentations, and 10 minutes for Q

- All presentations will be recorded and made available under Creative
  Commons licenses. In the Submission notes field, please indicate that
  you agree that your presentation will be licensed under the
  CC-By-SA-4.0 or CC-By-4.0 license and that you agree to have your
  presentation recorded. For example:

"If my presentation is accepted for FOSDEM, I hereby agree to
license all recordings, slides, and other associated materials under
the Creative Commons Attribution Share-Alike 4.0 International
License.  Sincerely, ."

- In the Submission notes field, please also confirm that if your talk
  is accepted, you will be able to attend FOSDEM and deliver your
  presentation. We will not consider proposals from prospective speakers
  who are unsure whether they will be able to secure funds for travel
  and lodging to attend FOSDEM. (Sadly, we are not able to offer travel
  funding for prospective speakers.)

---
Call for Volunteers
---

We are also looking for volunteers to help run the devroom. We need
assistance watching time for the speakers, and helping with video for
the devroom. Please contact Brian Proffitt (bkp at redhat.com), for more
information.

--
Questions?
--

If you have any questions about this devroom, please send your questions
to our devroom mailing list. 

iaas-virt-devr...@lists.fosdem.org

You can also subscribe to the list to receive updates about important
dates, session announcements, and to connect with other attendees.

===

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] devstack-plugin additional-pkg-repos: ocata summit working session?

[Kashyap Chamarthy]

On Tue, Oct 11, 2016 at 05:53:24PM +0200, Thomas Goirand wrote:
> On 10/11/2016 02:09 PM, Markus Zoeller wrote:
> > * How to create a "*.deb" package out of the source code of
> > libvirt/qemu? (surprisingly enough, I'm still struggling with this)
> 
> What version of libvirt / qemu are you trying to build? libvirt 2.3.0
> was released 6 days ago, and uploaded to Debian unstable 3 days ago. I
> don't think you need to manage the packaging yourself

I think what Markus is talking about is to the ability to produce
packages from arbitrary Git commits to be able to test Nova with certain
new features from libvirt / QEMU which won't be available until a formal
release is pushed out.

Somewhat analogus to the Fedora virt-preview-repository[1]:

"The Virtualization Preview Repository is for people who would like
to test the very latest virtualization related packages. This
repository is intended primarily as an aid to testing / early
experimentation. It is not intended for 'production' deployment."


[1] https://fedoraproject.org/wiki/Virtualization_Preview_Repository

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] devstack-plugin additional-pkg-repos: ocata summit working session?

[Kashyap Chamarthy]

On Tue, Oct 11, 2016 at 02:09:33PM +0200, Markus Zoeller wrote:
 
[Snip well-written backrgound detail]

> Request
> ---
> My question is, if you have interest in this plugin and its
> capabilities, are you at the Summit in Barcelona and do you have time
> for a short working session there? Maybe on Friday during the
> contributors meetup?
> 
> https://etherpad.openstack.org/p/ocata-nova-summit-meetup

I've made a note to be present there.

> Possible action/discussion items
> 
> * How to create a "*.deb" package out of the source code of
> libvirt/qemu? (surprisingly enough, I'm still struggling with this)

Probably check with some of the Debian packagers on the list?

FWIW, I'm familiar with producing RPM builds for libvirt / QEMU for
Fedora from their respective Git sources, as I make scratch builds for
testing often. 

> * How should we specify the packages and versions to install in the gate
> job?
> * When do we build new packages and increase the version in the gate?
> * Actually hack some code and push it

Another point for working there

   https://review.openstack.org/#/c/313568/4 -- Plugin to setup
   libvirt/QEMU from tar releases 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [libvirt] Debugging blockRebase() - "active block copy not ready for pivot"

[Kashyap Chamarthy]

On Fri, Oct 07, 2016 at 03:32:14PM -0500, Matt Riedemann wrote:
> On 10/6/2016 7:58 AM, Kashyap Chamarthy wrote:
> > -
> > We expose the state of the copy job in the XML and forward the READY
> > event from qemu to the users.
> > 
> > A running copy job exposes itself in the xml as:
> > 
> > 
> >   
> >   
> >   
> >   
> > 
> > 
> >   
> >   [...]
> > 
> > 
> > While the ready copy job is exposed as:
> > 
> > 
> >   
> >   
> >   
> >> ready='yes'>
> > 
> > 
> >   
> >   [...]
> > 

[...]

> Thanks for the great libvirtd log analysis, that's really helpful see what's
> going on and where we fail.

Took some time to write that, glad to know that at least one person has
read it :-)

> I've replied in Matthew's patch, which I think we can get in now regardless
> and backport.

Yes, noticed it just now.

> As for the fix, it sounds like mdbooth is going to work on the event
> listener code, which I'm hesitant to backport, but honestly this is such a
> latent broken flow that I don't think we really need to backport any fixes,
> at least for the event listener work to fix this long-term. The swap-volume
> test is disabled by default in Tempest and we enable it in devstack, so we
> can control which CI environments it runs in.
 
Agreed.


Also, maybe you have noticed, given Eric's reply on this thread (and the
upstream libvir-list), it is agreed that virDomainGetBlockJobInfo()
should be fixed "to quit reporting cur==end when ready:false".  This
allows the existing Nova code work correctly without any changes.

Discusion on Eric's reply in this thread:

http://lists.openstack.org/pipermail/openstack-dev/2016-October/105194.html

And upstream libvir-list

https://www.redhat.com/archives/libvir-list/2016-October/msg00347.html

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [libvirt] Debugging blockRebase() - "active block copy not ready for pivot"

[Kashyap Chamarthy]

On Thu, Oct 06, 2016 at 09:29:41AM -0500, Eric Blake wrote:
> On 10/06/2016 07:58 AM, Kashyap Chamarthy wrote:
> > On Thu, Oct 06, 2016 at 01:32:39AM +0200, Kashyap Chamarthy wrote:
> >> TL;DR
> >> -
> >>
> >> From the debug analysis of the log below, and discussion with Eric Blake
> >> of upstream QEMU / libvirt resulted in the below bug report:
> >>
> >>   https://bugzilla.redhat.com/show_bug.cgi?id=1382165 --
> >>   virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the
> >>   "ready" field of `query-block-jobs`
> > 
> > When I raised this on libvirt mailing list[0][1], one of the upstream
> > libvirt devs expressed an NACK in adjusting / "deliberately reporting
> > false data in block info structure".  Similar concern was also shared by
> > Matt Booth on #openstack-nova IRC.
> 
> I disagree with that sentiment. 

Interesting, I just noticed your argument here:

https://www.redhat.com/archives/libvir-list/2016-October/msg00249.html

> I think it is libvirt's responsibility to live up to libvirt's promise
> of virDomainGetBlockJobInfo() (namely, LIBVIRT documents that cur==end
> means the job is stable; and if qemu reports cur==end when the job is
> not stable, then it is libvirt that is lying to the upper user if it
> does NOT munge qemu's results to be accurate).
>
> As it is, we already patched libvirt to munge qemu's 0/0 into 0/1 when
> ready:false, so munging 123/123 into 122/123 when ready:false would just
> be another case of libvirt working around an infelicity of qemu.  There
> is NO INHERENT MEANING to the magnitude of cur and end, nor any
> requirement that end stays the same value across multiple calls to
> virDomainGetBlockJobInfo() - they are ONLY useful for a relative
> comparison of how much work remains to be done.  Munging the results IS
> appropriate.

Understood, this clarifies it, albeit a little messy. 

It indeed seems inconsistent to allow it in one case (like the one you
allude to above, fixed in 988218ca[1]) to adjust (& _not_ falsify, as
you accurately point out) libvirt reporting, but not the other case
(cur==end, "ready": false case when cur != 0).

So I re-opened the upstream libvirt bug[2] in light of your new
comments.
 

[1] http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=988218ca
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1382165 --
virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats &
the "ready" field of `query-block-jobs`

> That said, if you are going to work with existing libvirt that does
> not munge values, then yes, you either have to implement event
> handling or parse XML for the ready status, as existing libvirt's
> virDomainGetBlockJobInfo() is insufficient for the task.

Yep, noted.

Thanks for the great details, as usual.


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [libvirt] Debugging blockRebase() - "active block copy not ready for pivot"

[Kashyap Chamarthy]

On Thu, Oct 06, 2016 at 01:32:39AM +0200, Kashyap Chamarthy wrote:
> TL;DR
> -
> 
> From the debug analysis of the log below, and discussion with Eric Blake
> of upstream QEMU / libvirt resulted in the below bug report:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1382165 --
>   virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the
>   "ready" field of `query-block-jobs`

When I raised this on libvirt mailing list[0][1], one of the upstream
libvirt devs expressed an NACK in adjusting / "deliberately reporting
false data in block info structure".  Similar concern was also shared by
Matt Booth on #openstack-nova IRC.

Next, turns out the READY event is already exposed via the guest XML[1]:

-
We expose the state of the copy job in the XML and forward the READY
event from qemu to the users.

A running copy job exposes itself in the xml as:


  
  
  
  


  
  [...]


While the ready copy job is exposed as:


  
  
  
  


  
  [...]



Additionally we have anyncrhronous events that are emitted once qemu
notifies us that the block job has reached sync state or finished.
Libvirt uses the event to switch to the ready state.

The documentation suggests that block jobs should listen to the events
and act accordingly only after receiving the event.
-

So, Nova's is_job_complete() method & friends need to be reworked to
listen on the events for job readiness.

[0]
https://www.redhat.com/archives/libvir-list/2016-October/msg00217.html
[1] https://www.redhat.com/archives/libvir-list/2016-October/msg00229.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1382165#c3
 
> 
> Details
> ---
> 
> The code in Nova that's being executed is this part in _swap_volume()
> from libvirt/driver.py.
> 
> [...]
> # Start copy with VIR_DOMAIN_REBASE_REUSE_EXT flag to
> # allow writing to existing external volume file
> dev.rebase(new_path, copy=True, reuse_ext=True)
> 
> while not dev.is_job_complete():
> time.sleep(0.5)
>
> 
> dev.abort_job(pivot=True)
> [...]
> 

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] [libvirt] Debugging blockRebase() - "active block copy not ready for pivot"

[Kashyap Chamarthy]

TL;DR
-

>From the debug analysis of the log below, and discussion with Eric Blake
of upstream QEMU / libvirt resulted in the below bug report:

  https://bugzilla.redhat.com/show_bug.cgi?id=1382165 --
  virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the
  "ready" field of `query-block-jobs`

Until the above is fixed, I think _swap_volume() method in Nova could
either:

  (a) just try to either wait until it sees the BLOCK_JOB_READY event,
  which is emitted when a 'drive-mirror' job.  

  (b) Or keep polling, until we see "ready": true field (in the response
  of `query-block-jobs` inviked via libvirt blockJobInfo() 


Details
---

The code in Nova that's being executed is this part in _swap_volume()
from libvirt/driver.py.

[...]
# Start copy with VIR_DOMAIN_REBASE_REUSE_EXT flag to
# allow writing to existing external volume file
dev.rebase(new_path, copy=True, reuse_ext=True)

while not dev.is_job_complete():
time.sleep(0.5)

dev.abort_job(pivot=True)
[...]


(Optional Reading) libvirt / QEMU traffic log analysis 
--

Libvirt debug log (NB: 61MB)
http://logs.openstack.org/73/374373/2/check/gate-tempest-dsvm-full-ubuntu-xenial/149fe3e/logs/libvirt/libvirtd.txt.gz
Failed test: 
tempest.api.compute.admin.test_volume_swap.TestVolumeSwap.test_volume_swap

(I line-wrapped the logs manually.)

(1) We see _swap_volume() invokes the Libvirt blockRebase(), which calls
'drive-mirror' (ID: libvirt-25)

-
2016-10-04 15:54:45.308+: 30554: debug : qemuMonitorJSONCommandWithFd:291 :
Send command
'{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk1","target":"/dev/disk/by-path
/ip-10.205.221.168:3260-iscsi-iqn.2010-10.org.openstack:volume-7b86913d-78e4-4b68-8df2-63f1a4c6656a-lun-1","sync":"full","mode":"existing","format":"raw"},"id":"libvirt-25"}'
for write with FD -1
-


(2.a) Then, libvirt queries QEMU for job status (ID: 'libvirt-26') via
`query-block-jobs` command:

-
2016-10-04 15:54:45.332+: 30550: info : qemuMonitorIOWrite:528 :
QEMU_MONITOR_IO_WRITE: mon=0x7fa43000ee60
buf={"execute":"query-block-jobs","id":"libvirt-26"}
-

(2.b) Response for libvirt-26.  (Here we see: "len" == 1073741824,
"offset" == 0.  The copy job has just begun).

-
2016-10-04 15:54:45.332+: 30550: info :
qemuMonitorJSONIOProcessLine:206 : QEMU_MONITOR_RECV_REPLY:
mon=0x7fa43000ee60 reply={"return": [{"io-status": "ok", "device":
"drive-virtio-disk1", "busy": true, "len": 1073741824, "offset": 0,
"paused": false, "speed": 0, "ready": false, "type": "mirror"}], "id":
"libvirt-26"}
-

[Libvirt keeps on querying for job status, via `query-block-jobs`]
[Snip three polling requests 27, 28, 29]


(3.a) Next query (ID: libvirt-30):

-
2016-10-04 15:54:50.060+: 30550: info : qemuMonitorIOWrite:528 :
QEMU_MONITOR_IO_WRITE: mon=0x7fa43000ee60
buf={"execute":"query-block-jobs","id":"libvirt-30"}
-

(3.b) Response for libvirt-30.  (Here we see: "len" == 1073741824,
"offset" == 734003200 in bytes.  Copy operation is still in progress.)

-
2016-10-04 15:54:50.061+: 30550: info :
qemuMonitorJSONIOProcessLine:206 : QEMU_MONITOR_RECV_REPLY:
mon=0x7fa43000ee60 reply={"return": [{"io-status": "ok", "device":
"drive-virtio-disk1", "busy": true, "len": 1073741824, "offset":
734003200, "paused": false, "speed": 0, "ready": false, "type":
"mirror"}], "id": "libvirt-30"}
-

[Snip four (IDs, 31, 32, 33, 34) more job status request / response
round-trips.]


(4.a) Query for job status again (ID: libvirt-35):

-
2016-10-04 15:54:50.060+: 30550: info : qemuMonitorIOWrite:528 :
QEMU_MONITOR_IO_WRITE: mon=0x7fa43000ee60
buf={"execute":"query-block-jobs","id":"libvirt-30"}
-

(4.b) In response, finally, we see len (1073741824) == offset (1073741824):

-
2016-10-04 15:54:52.076+: 30550: info :
qemuMonitorJSONIOProcessLine:206 : QEMU_MONITOR_RECV_REPLY:
mon=0x7fa43000ee60 reply={"return": [{"io-status": "ok", "device":
"drive-virtio-disk1", "busy": true, "len": 1073741824, "offset":
1073741824, "paused": false, "speed": 0, "ready": false, "type":
"mirror"}], "id": "libvirt-35"}
-

Above, the "ready" flag does not yet indicate 'true', which signals the
job has _really_ ended.


Drilling down further


(5) The abort + pivot failed (as Nova calls
"dev.abort_job(pivot=True)"):

---
2016-10-04 15:54:52.076+: 30555: debug :
qemuDomainObjExitMonitorInternal:1923 : Exited monitor (mon=0x7fa43000ee60
vm=0x7fa414002a70 name=instance-0008)

2016-10-04 15:54:52.076+: 30555: error : qemuDomainBlockPivot:16137 : block
copy still active: disk 'vdb' not ready for pivot yet

2016-10-04 15:54:52.076+: 30555: debug : qemuBlockJobSyncEnd:242 : disk=vdb

2016-10-04 15:54:52.076+: 30555: debug : qemuDomainObjEndJob:1831 :
Stopping job: modify (async=none vm=0x7fa414002a70 name=instance-0008)
---

(6) HOWEVER, *here* we see 

Re: [openstack-dev] [nova][stable/liberty] Backport impasse: "virt: set address space & CPU time limits when running qemu-img"

[Kashyap Chamarthy]

On Tue, Sep 20, 2016 at 12:48:49PM +0200, Kashyap Chamarthy wrote:
> The said patch in question fixes a CVE[x] in stable/liberty.
> 
> We currently have two options, both of them have caused an impasse with
> the Nova upstream / stable maintainers.  We've had two-ish months to
> mull over this.  I'd prefer to get this out of a limbo, & bring this to
> a logical conclusion.
> 
> The two options at hand:
> 
> (1) Nova backport from master (that also adds a check for the presence
> of 'ProcessLimits' attribute which is only present in
> oslo.concurrency>=2.6.1; and a conditional check for 'prlimit'
> parameter in qemu_img_info() method.)
> 
> https://review.openstack.org/#/c/327624/ -- "virt: set address space
> & CPU time limits when running qemu-img"

Conclusion: After discussion and analysis on this thread, especially
Tony's response here[*], we went the route of option (1) above, and it
is now merged in stable/liberty


http://git.openstack.org/cgit/openstack/nova/commit/?h=stable/liberty=6bc37dc

Jeremy said (on #openstack-stable) he's going to follow up on the bug
for the security advisory.

Thanks everyone!

[*]
http://lists.openstack.org/pipermail/openstack-dev/2016-September/104303.html

> (2) Or bump global-requirements for 'oslo.concurrency'
> 
> https://review.openstack.org/#/c/337277/5 -- Bump
> 'global-requirements' for 'oslo.concurrency' to 2.6.1
> 
> Both patches have had long (and useful) discussion about their merits /
> demerits in the review comments in context of stable backports.  If you
> have sometime, I'd recommend going through the comments in both the
> reviews provides all the context, current disagreements.
> 
> 
> 
> [x] https://bugs.launchpad.net/nova/+bug/1449062 -- 
> qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][stable/liberty] Backport impasse: "virt: set address space & CPU time limits when running qemu-img"

[Kashyap Chamarthy]

On Thu, Sep 22, 2016 at 04:25:00PM +1000, Tony Breeds wrote:
> On Wed, Sep 21, 2016 at 02:05:51PM -0400, Sean Dague wrote:
> 
> > Well, the risk profile of what has to be changed for stable/liberty
> > (given that all the actual code is buried in libraries which have tons
> > of other changes). Special cherry-picked library versions would be
> > needed to fix this without openning up a ton of risk for breaking
> > stable/liberty badly.
> > 
> > That is the bit of work that no one seems to really have picked up.
> 
> So to clearly describe the work you touch on here:
> 
> We have:
>  * global-requirements.txt:
> origin/stable/liberty : oslo.concurrency>=2.3.0 # Apache-2.0
>  * upper-constraints.txt
> origin/stable/liberty : oslo.concurrency===2.6.1
>  * compatible oslo.concurrency releases:
> 2.3.0, 2.4.0, 2.5.0, 2.6.0 and 2.6.1(patched)
> 
> So to be sure that all liberty consumers have a reasonably simple update we'd
> need to create:
> 2.3.1, 2.4.1 and 2.5.1
> releases of oslo.concurrency
> 
> To achieve this we'd need to create a 3 short lived feature branches in
> oslo.concurrency and (I'm guessing) some CI changes so we can merge/release
> these versions.
> 
> We'd also need to update global-requirements.txt to:
> oslo.concurrency>=2.3.1,!=2.4.0,!=2.5.0,!=2.6.0
> 
> This update would be propagated to all projects:
> 
> Package  : oslo-concurrency [oslo-concurrency>=2.3.0] (used by 30 
> projects)
> Re-Release   : 5 projects
> Included in  : 17 projects
> Also affects : 8 projects
> 
> (The expanded list is at http://paste.openstack.org/show/582499/)
> 
> I haven't looked into the state of the libraries that need a re-release, but 
> my
> guess is that they'll have knock on effects if we're going to do this 
> properly.
> 
> There's a reason we call this kind of thing a tangled web of onions.
> 
> I suppose the most flexible solution would be to:
> 
> 1. Release the .1 versions
> 2. Leave global-requirements.txt
> 2. Add the patch to nova to with with/without the fix
> 3. Write appropriate words in the OSSN/OSSA
> 
> That gives deployers plenty of packages they can work with and public 
> backports
> of the fixes.  Yes g-r would be sub-optimal but the only thing that needs the
> fix will function with any of the versions listed.

Thanks for the succint analysis, Tony, that gives a clear view of state
of affairs if we had to go with three short-lived .1 releases.  Probably
this thread needs to be referred to in the commit message.

>  Time passes 
> 
> So I checked and the cherry-picks to 2.[345].0 are trivial so I think
> I just talked myself around to taking the nova fix now we can decide
> if we do all the .1 releases later

So, given your above comment, we seem to be going with the "option (1)"
as mentioned in the original post[x] -- i.e.  merge the Nova backport.
 
Thanks for the review (and the test with 2.6.0)

[x] 
http://lists.openstack.org/pipermail/openstack-dev/2016-September/104091.html

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] ops meetup feedback

[Kashyap Chamarthy]

On Tue, Sep 20, 2016 at 04:20:49PM +0100, Daniel P. Berrange wrote:
> On Tue, Sep 20, 2016 at 11:01:23AM -0400, Sean Dague wrote:

[...]

> > Here is my reconstruction of the snapshot issue from what I can remember
> > of the conversation.
> > 
> > Nova defaults to live snapshots. This uses the libvirt facility which
> > dumps both memory and disk. And then we throw away the memory. For large
> > memory guests (especially volume backed ones that might have a fast path
> > for the disk) this leads to a lot of overhead for no gain. The
> > workaround got them past it.
> 
> I think you've got it backwards there.
> 
> Nova defaults to *not* using live snapshots:
> 
> cfg.BoolOpt(
> 'disable_libvirt_livesnapshot',
> default=True,
> help="""
> Disable live snapshots when using the libvirt driver.
> ...""")
>
>
> When live snapshot is disabled like this, the snapshot code is unable
> to guarantee a consistent disk state. So the libvirt nova driver will
> stop the guest by doing a managed save (this saves all memory to
> disk), then does the disk snapshot, then restores the managed saved
> (which loads all memory from disk).
> 
> This is terrible for multiple reasons
> 
>   1. the guest workload stops running while snapshot is taken
>   2. we churn disk I/O saving & loading VM memory
>   3. you can't do it at all if host PCI devices are attached to
>  the VM
> 
> Enabling live snapshots by default fixes all these problems, at the
> risk of hitting the live snapshot bug we saw in the gate CI but never
> anywhere else.

Yes, FWIW, I agree.  In addition to the nice details above, enabling the
live snapshots also allows you to quiesce file systems for consistent
disk state, via the Glance image metadata properties
'hw_qemu_guest_agent' and 'os_require_quiesce'.  (Current cold snapshot
mechanism doesn't allow this.)


Anyhow, Sean seems to have submitted the change to toggle the config for
enabling live_snapshots:

https://review.openstack.org/#/c/373430/ -- "Change
disable_live_snapshot workaround"


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][stable/liberty] Backport impasse: "virt: set address space & CPU time limits when running qemu-img"

[Kashyap Chamarthy]

On Tue, Sep 20, 2016 at 11:57:26AM +0100, Daniel P. Berrange wrote:
> On Tue, Sep 20, 2016 at 12:48:49PM +0200, Kashyap Chamarthy wrote:

[...]

> > The two options at hand:
> > 
> > (1) Nova backport from master (that also adds a check for the presence
> > of 'ProcessLimits' attribute which is only present in
> > oslo.concurrency>=2.6.1; and a conditional check for 'prlimit'
> > parameter in qemu_img_info() method.)
> > 
> > https://review.openstack.org/#/c/327624/ -- "virt: set address space
> > & CPU time limits when running qemu-img"
> > 
> > (2) Or bump global-requirements for 'oslo.concurrency'
> > 
> > https://review.openstack.org/#/c/337277/5 -- Bump
> > 'global-requirements' for 'oslo.concurrency' to 2.6.1
> 
> Actually we have 3 options
> 
>   (3) Do nothing, leave the bug unfixed in stable/liberty

That was the unspoken third option, thanks for spelling it out. :-)

> While this is a security bug, it is one that has existed in every
> single openstack release ever, and it is not a particularly severe
> bug. Even if we fixed in liberty, it would still remain unfixed in
> every release before liberty. We're in the verge of releasing Newton
> at which point liberty becomes less relevant. So I question whether it
> is worth spending more effort on dealing with this in liberty
> upstream.  Downstream vendors still have the option to do either (1)
> or (2) in their own private branches if they so desire, regardless of
> whether we fix it upstream.

Sure, I agree with what you said.  This patch started off 2-ish months
ago, at that time it wasn't the "verge of releasing Newton".  That said,
if upstream feels it's not really necessary to get this into Liberty,
then I'm fine abandoning it, and close this.  That's at least brings
this to a conclusion.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][stable/liberty] Backport impasse: "virt: set address space & CPU time limits when running qemu-img"

[Kashyap Chamarthy]

The said patch in question fixes a CVE[x] in stable/liberty.

We currently have two options, both of them have caused an impasse with
the Nova upstream / stable maintainers.  We've had two-ish months to
mull over this.  I'd prefer to get this out of a limbo, & bring this to
a logical conclusion.

The two options at hand:

(1) Nova backport from master (that also adds a check for the presence
of 'ProcessLimits' attribute which is only present in
oslo.concurrency>=2.6.1; and a conditional check for 'prlimit'
parameter in qemu_img_info() method.)

https://review.openstack.org/#/c/327624/ -- "virt: set address space
& CPU time limits when running qemu-img"

(2) Or bump global-requirements for 'oslo.concurrency'

https://review.openstack.org/#/c/337277/5 -- Bump
'global-requirements' for 'oslo.concurrency' to 2.6.1

Both patches have had long (and useful) discussion about their merits /
demerits in the review comments in context of stable backports.  If you
have sometime, I'd recommend going through the comments in both the
reviews provides all the context, current disagreements.



[x] https://bugs.launchpad.net/nova/+bug/1449062 -- 
qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Bug team meeting

[Kashyap Chamarthy]

On Tue, Sep 06, 2016 at 03:41:04PM -0700, Augustina Ragwitz wrote:
> I should add that I also expressed interest in taking over the Bug Czar
> role. I've been actively involved with the bugs team since Markus
> revived it earlier this year. I've run meetings when he was unavailable
> and also provided assistance to new folks. How is it normally handled if
> more than one person is interested in taking on a subteam "czar" type of
> role? Should we co-czar?

While not speaking for Timofey, IMHO, sharing the role by two people in
different timezones is always useful in spreading the workload.


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] libvirt/qemu source install plugin.

[Kashyap Chamarthy]

On Thu, Jul 21, 2016 at 02:25:46PM +0200, Markus Zoeller wrote:
> On 20.07.2016 22:38, Mooney, Sean K wrote:
> > Hi
> > I recently had the need to test a feature (vhost-user reconnect)
> > that was commit to the qemu source tree a few weeks ago. As there
> > has been no release since then I needed to build from source so to
> > that end I wrote a small devstack plugin to do just that.
> > 
> > I was thinking of opening a review to create a new repo to host the
> > plugin under The openstack namespace
> > (openstack/devstack-plugin-libvirt-qemu) but before I do I wanted to
> > ask if others are interested In a devstack plugin that just compiles
> > and installs qemu and Libvirt?
> > 
> > Regards Sean.
> > 
> 
> tonby and I try to make the devstack plugin "additional package repos"
> (apr) work [1]. What you did is within the scope of that project. We
> also have an experimental job
> "gate-tempest-dsvm-nova-libvirt-kvm-apr"[2].  The last time I worked
> on this I wasn't able to create installable *.deb packages from
> libvirt + qemu source code. Other work items did then get more
> important and I had to pause the work on that.  I think we can work
> together to combine our efforts there.

NB: There's also in-progress work to allow configuring libvirt / QEMU
from source tar balls, as an external DevStack plugin:

https://review.openstack.org/#/c/313568/ -- Plugin to setup
libvirt/QEMU from tar releases

It was originally proposed (now abandoned, in favour of the above) as a
patch to DevStack proper, but was abandoned, as it was suggested to make
it as external plugin:

https://review.openstack.org/#/c/108714/

> 
> References:
> [1] https://github.com/openstack/devstack-plugin-additional-pkg-repos/
> [2]
> https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L565-L595
> 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][nova][infra] Voting gates

[Kashyap Chamarthy]

On Thu, Jul 14, 2016 at 03:19:39PM -0500, Matt Riedemann wrote:
> On 7/14/2016 11:34 AM, Paul Bourke wrote:
> > Hi Matt,
> > 
> > Here is the failure from nova_compute on trying to start an instance:

[...]

> > 2016-07-13 18:04:12.634968 | 2016-07-13 18:01:34.560 1 ERROR
> > oslo_service.service migration_flags &=
> > ~libvirt.VIR_MIGRATE_AUTO_CONVERGE
> > 2016-07-13 18:04:12.635022 | 2016-07-13 18:01:34.560 1 ERROR
> > oslo_service.service AttributeError: 'module' object has no attribute
> > 'VIR_MIGRATE_AUTO_CONVERGE'
> > 
> > The full log can be viewed at
> > http://logs.openstack.org/76/339776/7/check/gate-kolla-dsvm-deploy-ubuntu-source/0849a74/console.html#_2016-07-13_18_04_12_526704

[...]

> > > What version of libvirt/qemu do you have in the image/job you're running?
> > > 
> > > See:
> > > 
> > > https://github.com/openstack/nova/blob/92a388a1e34559b2ce69d31fdef996ff029495a6/nova/virt/libvirt/driver.py#L278
> > > 
> > > If you have libvirt>=1.2.3 and qemu>=1.6.0 then it's going to try and
> > > get these values from libvirt:
> > > 
> > > https://github.com/openstack/nova/blob/92a388a1e34559b2ce69d31fdef996ff029495a6/nova/virt/libvirt/driver.py#L633

Looking at the versions noted by Paul[1] in the upthread, they've
satisfied the above requirements.

> > > 
> > > Could be something patched out of the versions you're using from the
> > > distro maybe?

Yeah, I too wonder this could be it, because, see [*]
> > > 
> > > The actual failure paste/trace would help.
> > > 

[...]

> Looks like you have the minimum versions of libvirt and qemu required for
> VIR_MIGRATE_AUTO_CONVERGE but for whatever reason it's not actually in
> libvirt in the image you're testing against.
> 
> I'm not sure it would matter, but what version of libvirt-python is being
> used?

[*]

I extracted the contents of the said python-libvirt .deb from here:

$ wget 
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/mitaka-staging/+files/python-libvirt_1.3.1-1ubuntu1~cloud0_amd64.deb
$ ar vx python-libvirt_1.3.1-1ubuntu1~cloud0_amd64.deb
$ tar -xvf  data.tar.xz

And you do see VIR_MIGRATE_AUTO_CONVERGE attribute as part of
virDomainMigrateFlags():

$ cd usr/
$ grep -r VIR_MIGRATE_AUTO_CONVERGE
lib/python2.7/dist-packages/libvirt.py:VIR_MIGRATE_AUTO_CONVERGE = 8192


So, the said attribute is certainly present in the python-libvirt version
they're using.

> I'd probably ping danpb or kashyap about this in #openstack-nova in IRC.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][infra][ci] bulk repeating a test job on a single review in parallel ?

[Kashyap Chamarthy]

On Fri, Jul 01, 2016 at 02:35:34PM +, Jeremy Stanley wrote:
> On 2016-07-01 15:39:10 +0200 (+0200), Kashyap Chamarthy wrote:
> > [Snip description of some nice debugging.]
> > 
> > > I'd really love it if there was
> > > 
> > >  1. the ability to request checking of just specific jobs eg
> > > 
> > >   "recheck gate-tempest-dsvm-multinode-full"
> > 
> > Yes, this would really be desirable.  I recall once asking this exact
> > question on #openstack-infra, but can't find Infra team's response to
> > that.
> 
> The challenge here is that you want to make sure it can't be used to
> recheck individual jobs until you have them all passing (like
> picking a pin and tumbler lock). The temptation to recheck-spam
> nondeterministically failing changes is already present, but this
> would make it considerably easier still for people to introduce new
> nondeterministic failures in projects. Maybe if it were tied to a
> special pipeline type, and then we set it only for the experimental
> pipeline or something?

If it reduces nondeterministic spam for the CI Infra, and makes us
achieve the task at hand, sure.  [/me need to educate himself a
bit more on the Zuul pipeline infrastructure.]

Worth filing this (and your 'idle pipeline' thought below) in the Zuul
tracker here?

https://storyboard.openstack.org/#!/project/679

> > >  2. the ability to request this recheck to run multiple
> > > times in parallel. eg if i just repeat the 'recheck'
> > > command many times on the same patchset # without
> > > waiting for results
> > 
> > Yes, this too, would be _very_ useful for all the reasons you described.
> [...]
> 
> In the past we've discussed the option of having an "idle pipeline"
> which repeatedly runs specified jobs only when there are unused
> resources available, so that it doesn't significantly cut into our
> resource pool when we're under high demand but still allows to
> automatically collect a large amount of statistical data.
> 
> Anyway, hopefully James Blair can weigh in on this, since Zuul is
> basically in a feature freeze for a while to limit the number of
> significant changes we'll need to forward-port into the v3 branch.
> We'd want to discuss these new features in the context of Zuul v3
> instead.

> -- 
> Jeremy Stanley
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][infra][ci] bulk repeating a test job on a single review in parallel ?

[Kashyap Chamarthy]

On Thu, Jun 30, 2016 at 02:44:36PM +0100, Daniel P. Berrange wrote:

[Snip description of some nice debugging.]

> I'd really love it if there was
> 
>  1. the ability to request checking of just specific jobs eg
> 
>   "recheck gate-tempest-dsvm-multinode-full"

Yes, this would really be desirable.  I recall once asking this exact
question on #openstack-infra, but can't find Infra team's response to
that.

>  2. the ability to request this recheck to run multiple
> times in parallel. eg if i just repeat the 'recheck'
> command many times on the same patchset # without
> waiting for results

Yes, this too, would be _very_ useful for all the reasons you described.

Thanks for starting this thread!

> Any one got any other tips for debugging highly non-deterministic
> bugs like this which only hit perhaps 1 time in 100, without wasting
> huge amounts of CI resource as I'm doing right now ?
> 
> No one has ever been able to reproduce these failures outside of
> the gate CI infra, indeed certain CI hosting providers seem worse
> afffected by the bug than others, so running tempest locally is not
> an option.


[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] A primer on data structures used by Nova to represent block devices

[Kashyap Chamarthy]

On Thu, Jun 16, 2016 at 12:48:18PM +0100, Matthew Booth wrote:
> The purpose of this mail is to share what I have learned about the various
> data structures used by Nova for representing block devices. I compiled
> this for my own use, but I hope it might be useful for others, and that
> other might point out any errors.

Definitely!  Thanks for taking time to write this essay.

[Since you made the effort, worth submitting this to
nova/doc/source/nova-block-internals.rst (or some such).]

> As is usual when I'm reading code like this, I've created some cleanup
> patches to address nits or things I found confusing as I went along. I've
> posted review links at the end.
> 
> A note on reading this. I refer to local disks and volumes. A local disk in
> this context is any disk directly managed by nova compute. If nova is
> configured to use Rbd or NFS for instance disks these disks won't actually
> be local, but they are still managed locally and referred to as local disks.
> 
> There are 4 relevant data structures. 2 of these are general, 2 are
> specific to the libvirt driver.
> 
> BlockDeviceMapping
> ===
> 
> The 'top level' data structure is the block device mapping object. It is a
> NovaObject, persisted in the db. Current code creates a BDM object for
> every disk associated with an instance, whether it is a volume or not. I
> can't confirm (or deny) that this has always been the case, though, so
> there may be instances which still exist which have some BDMs missing.
> 
> The BDM object describes properties of each disk as specified by the user.
> It is initially created by the user and passed to compute api. Compute api
> transforms and consolidates all BDMs to ensure that all disks, explicit or
> implicit, have a BDM, then persists them.

What could be an example of an "implicit disk"?

> Look in nova.objects.block_device
> for all BDM fields, but in essence they contain information like
> (source_type='image', destination_type='local', image_id='),
> or equivalents describing ephemeral disks, swap disks or volumes, and some
> associated data.
> 
> Reader note: BDM objects are typically stored in variables called 'bdm'
> with lists in 'bdms', although this is obviously not guaranteed (and
> unfortunately not always true: bdm in libvirt.block_device is usually a
> DriverBlockDevice object). This is a useful reading aid (except when it's
> proactively confounding), as there is also something else typically called
> 'block_device_mapping' which is not a BlockDeviceMapping object.

[...]
 
> Reader beware: common usage is to pull 'block_device_mapping' out of this
> dict into a variable called 'block_device_mapping'. This is not a
> BlockDeviceMapping object, or list of them.
> 
> Reader beware: if block_device_info was passed to the driver by compute
> manager, it was probably generated by _get_instance_block_device_info(). By
> default, this function filters out all cinder volumes from
> block_device_mapping which don't currently have connection_info. In other
> contexts this filtering will not have happened, and block_device_mapping
> will contain all volumes.
> 
> Reader beware: unlike BDMs, block_device_info does not represent all disks
> that an instance might have. Significantly, it will not contain any
> representation of an image-backed local disk, i.e. the root disk of a
> typical instance which isn't boot-from-volume. Other representations used
> by the libvirt driver explicitly reconstruct this missing disk. I assume
> other drivers must do the same.

[Meta comment: Appreciate these "Reader beaware"s -- they're having
the right effect -- causing my brain to 'stand up and read more than
twice' to assimilate.]

 
> instance_disk_info
> =
> 
> The driver api defines a method get_instance_disk_info, which returns a
> json blob. The compute manager calls this and passes the data over rpc
> between calls without ever looking at it. This is driver-specific opaque
> data. It is also only used by the libvirt driver, despite being part of the
> api for all drivers. Other drivers do not return any data. The most
> interesting aspect of instance_disk_info is that it is generated from the
> libvirt XML, not from nova's state.
> 
> Reader beware: instance_disk_info is often named 'disk_info' in code, which
> is unfortunate as this clashes with the normal naming of the next
> structure. Occasionally the two are used in the same block of code.
> 
> instance_disk_info is a list of dicts for some of an instance's disks.

The above sentence reads a little awkward (maybe it's just me), might
want to rephrase it if you're submitting it as a Gerrit change.

While reading this section, among other places, I was looking at:
_get_instance_disk_info() ("Get the non-volume disk information from the
domain xml") from nova/virt/libvirt/driver.py.

> Reader beware: Rbd disks (including non-volume disks) and cinder volumes
> are not included in instance_disk_info.
> 
> The dicts are:
> 
>   {
>   

Re: [openstack-dev] [gate] [nova] live migration, libvirt 1.3, and the gate

[Kashyap Chamarthy]

On Thu, May 26, 2016 at 10:55:47AM -0400, Sean Dague wrote:
> On 05/26/2016 05:38 AM, Kashyap Chamarthy wrote:
> > On Wed, May 25, 2016 at 05:42:04PM +0200, Kashyap Chamarthy wrote:
> > 
> > [...]
> > 
> >> So, in short, the central issue seems to be this: the custom 'gate64'
> >> model is not being trasnalted by libvirt into a model that QEMU can
> >> recognize.
> > 
> > An update:
> > 
> > Upstream libvirt points out that this turns to be regression, and
> > bisected it to commit (in libvirt Git): 1.2.9-31-g445a09b -- "qemu:
> > Don't compare CPU against host for TCG".
> > 
> > So, I expect there's going to be fix pretty soon upstream libvirt.
> 
> Which is good... I wonder how long we'll be waiting for that back in our
> distro packages though.

Yeah, until the fix lands, our current options seem to be:

  (a) Revert to a known good version of libvirt
  
  (b) Use nested virt (i.e. ) -- I doubt is possible
  on RAX environment, which is using Xen, last I know.
  
  (c) Or a different CPU model


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] [nova] live migration, libvirt 1.3, and the gate

[Kashyap Chamarthy]

On Wed, May 25, 2016 at 05:42:04PM +0200, Kashyap Chamarthy wrote:

[...]

> So, in short, the central issue seems to be this: the custom 'gate64'
> model is not being trasnalted by libvirt into a model that QEMU can
> recognize.

An update:

Upstream libvirt points out that this turns to be regression, and
bisected it to commit (in libvirt Git): 1.2.9-31-g445a09b -- "qemu:
Don't compare CPU against host for TCG".

So, I expect there's going to be fix pretty soon upstream libvirt.

> I could reproduce it with upstream libvirt
> (libvirt-1.3.4-2.fc25.x86_64), and filed this bug:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1339680 -- libvirt CPU
> driver fails to translate a custom CPU model into something that
> QEMU recognizes

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] [nova] live migration, libvirt 1.3, and the gate

[Kashyap Chamarthy]

On Tue, May 24, 2016 at 01:59:17PM -0400, Sean Dague wrote:

Thanks for the summary, Sean.

[...]

> It turns out it works fine because libvirt *actually* seems to take the
> data from cpu_map.xml and do a translation to what it believes qemu will
> understand. On these systems apparently this turns into "-cpu
> Opteron_G1,-pse36"
> (http://logs.openstack.org/29/42529/24/check/gate-tempest-dsvm-multinode-full/5f504c5/logs/libvirt/qemu/instance-000b.txt.gz)
> 
> At some point between libvirt 1.2.2 and 1.3.1, this changed. Now libvirt
> seems to be passing our cpu_model directly to qemu, and assumes that as
> a user you will be responsible for writing all the  stanzas to
> add/remove yourself. When libvirt sends 'gate64' to qemu, this explodes,
> as qemu has no idea what we are talking about.
> http://logs.openstack.org/34/319934/2/experimental/gate-tempest-dsvm-multinode-live-migration/b87d689/logs/screen-n-cpu.txt.gz#_2016-05-24_15_59_12_531

[...]

So, in short, the central issue seems to be this: the custom 'gate64'
model is not being trasnalted by libvirt into a model that QEMU can
recognize.

I could reproduce it with upstream libvirt
(libvirt-1.3.4-2.fc25.x86_64), and filed this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1339680 -- libvirt CPU
driver fails to translate a custom CPU model into something that
QEMU recognizes

Some discussion from libvirt migration developers (comment #3):

"So it looks like the whole code which computes the right CPU model
is skipped. The reason is . Our code avoids
comparing guest CPU definition to host CPU for TCG mode (since the
host CPU is irrelevant in this case). And as a side effect the code
that would translate the gate64 CPU model into something that is
supported by QEMU is skipped too."

> So, the existing cpu_map.xml workaround for our testing situation will
> no longer work.

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] - suggested development workflow without ./rejoin-stack.sh ?

[Kashyap Chamarthy]

On Mon, May 02, 2016 at 03:10:56PM -0700, Kevin Benton wrote:
> This patch removed the ./rejoin-stack.sh script:
> https://review.openstack.org/#/c/291453/
> 
> I relied on this heavily in my development VM which sees lots of restarts
> because of various things (VM becomes unresponsive in load testing, my
> laptop has a kernel panic, etc). Normally this was not a big deal because I
> could ./rejoin-stack.sh and pick up where I left off (all db objects,
> virtual interfaces, instance images, etc all intact).
> 
> Now am I correct in understanding that when this happens there is no way to
> restart the services in a simple manner without blowing away everything and
> starting over? Unless I'm missing some way to run ./stack.sh without losing
> previous state, this seems like a major regression (went from mostly
> working ./rejoin-stack.sh to nothing).
> 
> What is the recommended way to use devstack without being a power outage
> away from losing hours of work?

FWIW, whenever I feel I have a working env. in DevStack, I take a qcow2
live internal snapshot:

$ sudo virsh snapshot-create-as devstack \
snap1 "Working setup for bug#123"

If something goes terribly wrong in my env, revert to a known sane state:

$ virsh snapshot-revert devstack snap1

This stipulates[*] that you use Qcow2 format.  Also it does not exactly
solve your 'sudden power outage' issue, but comes reasonably close if
you save the work at points-in-time you care about.

[*] There are other methods like 'external disk snapshots' (when you
create a snapshot, the current disk becomes a 'backing file' & a new
qcow2 overlay is created to track all the new writes from the point
of taking snapshot) that allow you both Raw and Qcow2 file formats. 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][elections] Results of the TC Election

[Kashyap Chamarthy]

On Fri, Apr 08, 2016 at 03:14:33PM +0200, Thierry Carrez wrote:

[...]

> Long term I'd like to remove the summit pass perk (or no longer link
> it to "one commit").

Agreed.  FWIW, I recall discussing this in the hallways of the Paris
Summit, and suggesting something like 3-5 commits and someway to also
include aspects like meaningful investigative comments in bugs, root
cause analysis / providing reproducers, etc -- they're not sexy, and
won't result in commits, but damn sure are valuable, as we know.

> It will likely result in a drop in contributors numbers
> (gasp), but a saner electorate.

It certainly deters the 'do-just-enough-to-get-a-pass' people; 
if that means less contributors, so be it.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] working on bug reports; what blocks you?

[Kashyap Chamarthy]

On Fri, Mar 18, 2016 at 07:01:34PM +0100, Markus Zoeller wrote:
> Kashyap Chamarthy <kcham...@redhat.com> wrote on 03/18/2016 07:28:09 AM:

[...]

> > Writing a thoughtful report is hard and time-taking.
> 
> Yeah, and I assume that's the reason many bug reports lack that
> information. I have the hope to dig deeper into the logging
> capabilities of Nova during the P cycle, to figure out how much we
> internally already know but don't offer easily enough. 

As you know, we have some bits in place:

  - Guru Meditation Reports[1].  E.g. `kill -s USR1 `pgrep nova-compute`

  - General logging/tracing of API calls via 

  - For Virt drivers, server-side libvirt/QEMU logging.  We have the
relevant log filters enabled in DevStack[2], so Gate machines have
these enabled.  For external bugs being reported in this area,
people have to enable these explicitly and repeat their tests.

  - General logging/tracing of API calls via 'debug|verbose = True'

[1] http://docs.openstack.org/developer/nova/gmr.html
[2] 
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova_plugins/functions-libvirt#n104

> In some bug reports I suggested to use sosreport and attach the file
> then, but I didn't see that happen.

Yeah, that's a good idea, if people could remember to do that, as it
adds far more context from the system than merely OpenStack or its
immediate relevant logs, reducing the reporter <-> triager roundtrip.

> In my head there would be openstack CLI commands like these two:
> 
> $ openstack report-bug 
> $ openstack report-bug --last 10m 
> 
> That should then result in an upstream bug report which answers the
> usual questions we have. Don't ask me details how this would happen.

[...]
 
> In an ideal world with unlimited resources we could do that on our own
> without asking the reporter, but I guess we should do the "please 
> recheck if it's in master" thing more often.

We could certainly try.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] working on bug reports; what blocks you?

[Kashyap Chamarthy]

On Thu, Mar 17, 2016 at 03:28:48PM -0500, Matt Riedemann wrote:
> On 3/17/2016 11:41 AM, Markus Zoeller wrote:
> >What are the various reasons which block you to work on bug reports?
> >This question goes especially to the new contributors but also to the
> >rest of us. For me, personally, it's that most bug reports miss the
> >steps to reproduce which allow me to see the issue on my local system
> >before I start to dig into the code.
> >
> >I'm asking this because I'm not sure what the main reasons are that
> >our bug list is this huge (~1000 open bug reports). Maybe you have
> >reasons which can be resolved or mitigated by me in my bug czar role.
> >Let me know.

Effective bug reporting is top issue for me.  By "effective" I mean:

  - Not assuming any prior context while writing a report.  (Especially
when writing how to reproduce the problem.)
  - Not forgetting to state changes made to various configuration
attributes
  - Describing the problem's symptoms in chronological order.
  - Describing the test environment precisely.

Writing a thoughtful report is hard and time-taking.

https://wiki.openstack.org/wiki/BugFilingRecommendations

> Clear recreate steps is probably #1, but also logs if there are
> obvious failures. A stacktrace goes a long way with a clear
> description of the failure scenario. Obviously need to know the level
> of code being tested.
> 
> For a lot of bugs that are opened on n-2 releases, like kilo at this
> point, my first question is, have you tried this on master to see if
> it's still an issue. That's lazy on my part, but it's easy if I'm not
> aware of a fix that just needs backporting.

I don't view it as being lazy on your part.  Other open source projects
use a similar method -- e.g. in Fedora Project, one month after N+2
(Fedora-24) is released, 'N' (Fedora-22) goes End-of-Life.  And, all
bugs (that are not solved) reported against 'N' (for components with
high bug volume) are closed, with a request to re-test them on N+2
(latest stable release), and re-open it if the issue persists.
Otherwise, it becomes difficult to cope with volume.


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Wishlist bugs == (trivial) blueprint?

[Kashyap Chamarthy]

On Tue, Mar 15, 2016 at 05:59:32PM +, Tim Bell wrote:

[...]

> The bug process was very light weight for an operator who found
> something they would like enhanced. It could be done through the web
> and did not require git/gerrit knowledge. I went through the process
> for a change:
> 
> - Reported a bug for the need to add an L2 cache size option for QEMU
> (https://bugs.launchpad.net/nova/+bug/1509304) closed as invalid since
> this was a feature request - When this was closed, I followed the
> process and submitted a spec
> (https://blueprints.launchpad.net/nova/+spec/qcow2-l2-cache-size-configuration)
> 
> It was not clear how to proceed from here for me. 

Given the feature request is fairly self-contained (so, it wouldn't
warrant a specification), the next step would be someone feeling
motivated enough to submit a patch to implement it.

> The risk I see is that we are missing input to the development process
> in view of the complexity of submitting those requirements. Clearly,
> setting the bar too low means that there is no clear requirement
> statement etc. However, I think the combination of tools and
> assumption of knowledge of the process means that we are missing the
> opportunity for good quality input.

>From an Operator's perspective, I think your concern seems reasonable:
having a friction-free way to provide input (like an RFE bug) vs. having
the process knowledge (Spec-less Blueprint vs. Blueprint with Spec).

But, given Markus' stats about 'Wishlist' implementation, that category
doesn't seem to be quite effective.

[...]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Update on live migration priority

[Kashyap Chamarthy]

On Fri, Feb 12, 2016 at 04:21:27PM +, Murray, Paul (HP Cloud) wrote:
> This time with a tag in case anyone is filtering...

Yep, I was filtering, and would've missed it without your tag. :-)

> From: Murray, Paul (HP Cloud)
> Sent: 12 February 2016 16:16
> To: openstack-dev@lists.openstack.org
> Subject: [openstack-dev] Update on live migration priority
> 
> The objective for the live migration priority is to improve the
> stability of migrations based on operator experience. The high level
> approach is to do the following:
> 
> 1.   Improve CI
> 
> 2.   Improve documentation
>
> 
> 3.   Improve manageability of migrations
> 
> 4.   Fix bugs
> 
> In this cycle we targeted a few immediately implementable features
> that would help, specifically giving operators commands to allow them
> to manage migrations (inspect progress, force completion, and cancel)
> and improve security (split-networks and remove ssh-based
> resize/migration; aka storage pools).
> 
> Most of these are on track to be completed in this cycle with the
> exception of storage pools work which is being deferred. Further
> details follow.
> 
> Expand CI coverage - in progress
> 
> There is a job in the experimental queue called:
> gate-tempest-dsvm-multinode-live-migrationqueued. This will become the
> job that performs live migration tests; any live migration tests in
> other jobs will be removed. At present the job has been configured to
> cover different storage configurations including cinder, NFS, ceph.
> Tests are now being added to the job. Patches are currently up for
> live migration of instances with swap and instances with ephemeral
> disks.
> 
> Please trigger the experimental queue if your patches touch migrations
> in some way so we can check the stability of the jobs. Once stable and
> with sufficient tests we will promote the job from the experimental
> queue so that it always runs.
> 
> See: https://review.openstack.org/#/q/topic:lm_test
> 
> Improve API docs - done
> 
> Some changes were made to the API guide for moving servers, including
> better descriptions for the server actions migrate, live migrate,
> shelve, resize and evacuate (
> http://developer.openstack.org/api-guide/compute/server_concepts.html#server-actions
> ) and a section that describes reasons for moving VMs with common use
> cases outlined (
> http://developer.openstack.org/api-guide/compute/server_concepts.html#moving-servers
> )
> 
> Block live migration with attached volumes - done
> 
> The selective block device migration API in libvirt 1.2.17 is used to
> allow block migration when volumes are attached. A follow on patch to
> allow readonly drives to be copied in block migration has not been
> completed. This patch is required to allow iso9600 format config
> drives to be migrated. Without it only vfat config drives can be
> migrated. There is still some thought going into that - see:
> https://review.openstack.org/#/c/234659
> 
> Force complete - requires python-novaclient change
> 
> Force-complete forces a live migration to complete  by pausing the VM
> and restarting it when it has completed migration. This is intended as
> a brute force way to make a VM complete its migration when it is
> taking too long. In the future auto-converge and post-copy will be
> looked at. These became available in qemu 2.5.
> 
> Force complete is done in nova but still requires a change to
> python-novaclient to implement the CLI.
> 
> Cancel - in progress
> 
> Cancel stops a live migration, leaving it on the source host with the
> migration status left as "cancelled". This is in progress and follows
> the pattern of force-complete. Unfortunately this needs to be bundled
> up into one patch to avoid multiple API bumps.
> 
> Patches for review:
> https://review.openstack.org/#/q/status:open+topic:bp/abort-live-migration
> 
> Progress reporting - in progress (no pun intended)
> 
> Progress reporting introduces migrations as a sub-resource of servers
> and adds progress data to the migration record. There was some debate
> at the mid cycle and on the mailing list about how to record this
> transient data. It is a waste to keep writing it to the database, but
> as it is generated at the compute manager but examined at the API it
> was felt that writing it to the database is necessary to fit the
> existing architecture. The conclusions was that writing to the
> database every 5 seconds would not cause a significant overhead.
> Alternatives could be persued later if necessary. For discussion see
> this ML thread:
> http://lists.openstack.org/pipermail/openstack-dev/2016-February/085662.html
> and the IRC meeting transcript here:
> http://eavesdrop.openstack.org/meetings/nova_live_migration/2016/nova_live_migration.2016-02-09-14.01.log.html
> 
> Patches for review:
> https://review.openstack.org/#/q/status:open+topic:bp/live-migration-progress-report
> 
> Split networking - done
> 
> Split networking adds a configuration parameter to specify
> 

Re: [openstack-dev] [all][tc] Stabilization cycles: Elaborating on the idea to move it forward

[Kashyap Chamarthy]

On Thu, Jan 21, 2016 at 06:37:00PM +0100, Markus Zoeller wrote:
> Flavio Percoco  wrote on 01/21/2016 09:13:02 AM:

[...]

First, positive remark(s): 

Thanks for writing this up.  FWIW, I support the notion of having
milestones focusing on stability, as opposed to explicitly declaring a
whole cycle as 'stable' as I agree (as do you) with Dan Berrange's
reasoning.

(Also see Doug's comment in the thread: "projects the option of choosing
a release model that allows for multiple releases per 6 month cycle,
while still maintaining a single stable release after the cycle.")

> > >> Negative(?) effects
> > >> ===
> > >>
> > >> - Project won't get new features for a period of time Economic
> > >> impact  on developers(?)
> > >> - It was mentioned that some folks receive bonuses for landed 
> > >>   features

This (non-point) reminds of a recent comment I've read elsewhere[1]
about why websites have been becoming bloated[1], and how people are
(dis)incentivized.  [NB: This was written in the context of websites;
we're talking about an Infra project, so adjust the view accordingly]:

   "[...] People (designers, coders, etc) get bonuses and paychecks for
   creating stuff more than tearing down stuff.

   Put this on your resume -- "Implemented feature x, designed y, added
   z" vs "Cut out 10k lines worth of crap only 10% of customers
   [operators] used, stripped away stupid 1Mb worth for js that displays
   animated snowflakes, etc".  You'd produce a better perception by
   claiming you added / created / built, rather than deleted.

   So it is not surprising that more stuff gets built, more code added
   to the pile, more features implemented. Heck, even GMail keeps
   changing every 6 months for apparently no reason. But in reality
   there is a reason -- Google has full time designers on the GMail
   team. There is probably no way they'd end the year with "Yap, site
   worked great, we did a nice job 2 years ago, so I didn't touch it
   this year."

[...]
 
> I try to handle in one post the different aspects which came up so
> far:
> 
> wrt dedicated stabilization cycles|milestones:
> 
> Piled up (=older) bugs are harder to solve than fresh ones.  I've
> seen next to no bug report in Nova which has all the necessary
> data to do a proper analysis. There are usually 1-3 requests to
> the bug reporter necessary to get enough data.  This makes me
> believe that stabilization should be a continuous effort.

Whole-heartedly agree with this.  It just ought to be a _continuous_
effort.

While we're at it, thanks Markus for your patient (and productive)
efforts on bug triaging in Nova!

[...]

[1] https://news.ycombinator.com/item?id=10820716

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][infra] Ability to run newer QEMU in Gate jobs

[Kashyap Chamarthy]

On Wed, Jan 20, 2016 at 10:25:00AM +, Tony Breeds wrote:
> On Tue, Jan 19, 2016 at 06:11:35PM +, Daniel P. Berrange wrote:
> 
> > We'll almost certainly need to be able to test QEMU 2.6 in the N
> > release cycle, since that'll (hopefully) include support for TLS
> > encrypted migration & nbd traffic.  So I don't think waiting for
> > LTS releases is a viable strategy in general - we'll need UCA to
> > be available for at least some set of jobs we run. Alternatively
> > stick with LTS release for Ubuntu, and run other jobs with Fedora
> > and the virt-preview repository to give us coverage of the cutting
> > edge QEMU/libvirt stack.
> 
> I'm working on a devstack-plugin that will allow us to do these
> things.  Right now it's Ubuntu specific but nothing that'll stop F23
> once that's working in nodepool.

Good to know.

A related piece of work (no update over a year), that needs to be turned
into a plugin:

https://review.openstack.org/#/c/108714/10 -- Support for
libvirt/QEMU tar releases

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][infra] Ability to run newer QEMU in Gate jobs

[Kashyap Chamarthy]

Heya,

Currently the live migration test job[1] is using relatively old version
of QEMU (2.0 -- 2 years old, 17-APR-2014).  And, libvirt 1.2.2
(released on 02-MAR-2014).  For libvirt, I realize there's an
in-progress thread[2] to get to a state to run a bleeding edge libvirt.

How can we go about bumping up QEMU to its newest stable release (2.5,
DEC-2015)?

Matt Riedemann tells me on IRC that multi-node live migration job is
currently Ubuntu only, and to get a newer QEMU, it has to be added to
Ubuntu Cloud Archive.  It'd be useful to get that done.  Who can help
with it?  

It'll allow us to confirm our suspicion that a couple of Nova live
migration bugs[3][4] are likely fixed with that version.  Also it (the
newer QEMU) has some additional debug logging capabilities which can
help us with root cause analysis of some live migration issues.


[1] 
https://jenkins05.openstack.org/job/gate-tempest-dsvm-multinode-live-migration/
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-November/079679.html
[3] https://bugs.launchpad.net/nova/+bug/1524898 -- Volume based live
migrtion aborted unexpectedly
[4] https://bugs.launchpad.net/nova/+bug/1535232/ -- live-migration ci
failure on nfs shared storage 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] VIR_MIGRATE_NON_SHARED_INC works more like full copy in block migration ?

[Kashyap Chamarthy]

On Sun, Jan 10, 2016 at 06:07:33PM +0800, Luo Gangyi wrote:
> Hi devs,
> 
> 
> Do you test the difference between within and without
> VIR_MIGRATE_NON_SHARED_INC ?
> 
> 
> When I add VIR_MIGRATE_NON_SHARED_INC in block_migration_flags in
> nova, nova block migration behaves more like a full copy of disk.  The
> disk in destination nodes is large(10GB+) and the process of live
> migration is slow.
> 
> 
> However, when I remove  VIR_MIGRATE_NON_SHARED_INC in
> block_migration_flags in nova, nova block migration behaves more like
> a incremental copy of disk.  The disk in destination nodes is
> small(10MB-) and the process of live migration is very fast.
> 
> 
> So I was confused about what VIR_MIGRATE_NON_SHARED_INC really means.

There are two flags that are related, that might be helpful to be clear
about: VIR_MIGRATE_NON_SHARED_DISK and VIR_MIGRATE_NON_SHARED_INC.

Below is an explanation[1] with example, by Eric Blake, from
libvirt-users list.

NB: It is talking in terms of `virsh` commands, where
'--copy-storage-all' means VIR_MIGRATE_NON_SHARED_DISK and
'--copy-storage-inc' means VIR_MIGRATE_NON_SHARED_INC:

"If you use qcow2 backing chains for thin provisioning, as in:

base.qcow2 <- active.qcow2

then --copy-storage-all copies the entire disk over to the
destination (both base.qcow2 and active.qcow2 contents), while
--copy-storage-inc copies only active.qcow2 (and assumes you have
manually copied base.qcow2 in some other means - since it is
read-only, it won't change from the last migration, and there are
some storage arrays where you can very efficiently copy files
around).  Thus, --copy-storage-inc can be MUCH faster if
active.qcow2 represents a small delta in relation to base.qcow2."


[1] https://www.redhat.com/archives/libvirt-users/2015-April/msg00172.html
Semantics of "virsh migrate --copy-storage-all" vs.
--copy-storage-inc



-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][infra] Getting a bleeding edge libvirt gate job running

[Kashyap Chamarthy]

On Wed, Nov 18, 2015 at 06:46:58AM +1100, Ian Wienand wrote:
> On 11/18/2015 06:10 AM, Markus Zoeller wrote:
>  This
> >was a trigger to see if we can create a gate job which utilizes the
> >latest, bleeding edge, version of libvirt to test such features.
> 
> >* Is already someone working on something like that and I missed it?
> 
> I believe the closest we have got is probably [1]; pulling apart some
> of the comments there might be helpful

After months of review, and being almost close to merge, the "Support
for libvirt/QEMU tar releases" patch you refer to below, seems to be
abandoned by the contributor.

It is functionally fine and just ought to put in an external plugin (as
you note below) and add the relevant glue code in DevStack to call it.

> In short, a devstack plugin that installs the latest libvirt is
> probably the way to go.
>
> Ongoing, the only issue I see is that we do things to base devstack
> that conflict/break this plugin, as we are more-or-less assuming the
> distro version (see things like [2], stuff like this comes up every
> now and then).
> 
> >* If 'no', is there already a repo which contains the very latest
> >   libvirt builds which we can utilize?
> 
> For Fedora, there is virt-preview  [3] at least

Yep, it is actively maintained.

> 
> [1] https://review.openstack.org/#/c/108714/
> [2] https://review.openstack.org/246501
> [3] https://fedoraproject.org/wiki/Virtualization_Preview_Repository

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][bugs] Weekly Status Report

[Kashyap Chamarthy]

On Mon, Nov 09, 2015 at 04:55:12PM +0100, Markus Zoeller wrote:
> Kashyap Chamarthy <kcham...@redhat.com> wrote on 11/06/2015 06:37:08 PM:
> 
> > From: Kashyap Chamarthy <kcham...@redhat.com>
> > To: "OpenStack Development Mailing List (not for usage questions)" 
> > <openstack-dev@lists.openstack.org>
> > Date: 11/06/2015 06:37 PM
> > Subject: Re: [openstack-dev] [nova][bugs] Weekly Status Report
> > 
> > On Fri, Nov 06, 2015 at 05:54:59PM +0100, Markus Zoeller wrote:
> > > Hey folks,
> > > 
> > > below is the first report of bug stats I intend to post weekly.
> > > We discussed it shortly during the Mitaka summit that this report
> > > could be useful to keep the attention of the open bugs at a certain
> > > level. Let me know if you think it's missing something.
> > 
> > Nice.  Thanks for this super useful report (especially the queries)!
> > 
> > For cadence, I feel a week flies by too quickly, which is likely to
> > cause people to train their muscle memory to mark these emails as read.
> > Maybe bi-weekly?
> 
> How about having it weekly until the Mitaka-2 milestone in January
> and see how this works? 

Sure, sounds good.

> If it is perveived as more noise on the ML then I can change it to
> bi-weekly.



-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][bugs] Weekly Status Report

[Kashyap Chamarthy]

On Fri, Nov 06, 2015 at 05:54:59PM +0100, Markus Zoeller wrote:
> Hey folks,
> 
> below is the first report of bug stats I intend to post weekly.
> We discussed it shortly during the Mitaka summit that this report
> could be useful to keep the attention of the open bugs at a certain
> level. Let me know if you think it's missing something.

Nice.  Thanks for this super useful report (especially the queries)!

For cadence, I feel a week flies by too quickly, which is likely to
cause people to train their muscle memory to mark these emails as read.
Maybe bi-weekly?

> 
> New bugs which are *not* assigned to any subteam
> 
> count: 19
> query: http://bit.ly/1WF68Iu
> 
> 
> New bugs which are *not* triaged
> 
> subteam: libvirt 
> count: 14 
> query: http://bit.ly/1Hx3RrL
> subteam: volumes 
> count: 11
> query: http://bit.ly/1NU2DM0
> subteam: network : 
> count: 4
> query: http://bit.ly/1LVAQdq
> subteam: db : 
> count: 4
> query: http://bit.ly/1LVATWG
> subteam: 
> count: 67
> query: http://bit.ly/1RBVZLn
> 
> 
> High prio bugs which are *not* in progress
> --
> count: 39
> query: http://bit.ly/1MCKoHA
> 
> 
> Critical bugs which are *not* in progress
> -
> count: 0
> query: http://bit.ly/1kfntfk
> 
> 
> Readings
> 
> * https://wiki.openstack.org/wiki/BugTriage
> * https://wiki.openstack.org/wiki/Nova/BugTriage
> * 
> http://lists.openstack.org/pipermail/openstack-dev/2015-November/078252.html
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] live migration sub-team meeting

[Kashyap Chamarthy]

On Tue, Nov 03, 2015 at 12:18:38PM +, Murray, Paul (HP Cloud) wrote:
> Hi all,
> 
> Live migration was confirmed as a Nova priority for the Mitaka cycle
> and a sub-team section can be found on the priorities tracking page
> [1].
> 
> Most team members expressed they would like a regular IRC meeting for
> tracking work and raising blocking issues. Looking at the contributors
> here [2], most of the participants seem to be in the European
> continent (in time zones ranging from UTC to UTC+3) with a few in the
> US (please correct me if I am wrong). That suggests that a time around
> 1500 UTC makes sense.
> 
> I would like to invite suggestions for a day and time for a weekly
> meeting - 

Maybe you could create a quick Doodle poll to reach a rough consensus on
day/time:

http://doodle.com/

> you do not have to go by the above suggestion. When we have
> a time I will arrange the meeting and create a corresponding wiki
> page.
> 
> Please note that attendance is not a requirement for participation,
> but it will help to track or coordinate some activities.
> 
> Paul
> 
> [1]  https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking
> [2] https://etherpad.openstack.org/p/mitaka-live-migration
 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Change from Mitaka: Expected UNIX signal to generate Guru Meditation (error) Reports

[Kashyap Chamarthy]

On Sat, Oct 31, 2015 at 09:11:31AM +0900, Davanum Srinivas wrote:
> Matt,
> 
> 0.6.0 and 0.7.0 were released for Mitaka! not Liberty. Though yes, because
> of not having pins any library we release for Mitaka will end up being used
> with Liberty as well.
> http://lists.openstack.org/pipermail/openstack-announce/2015-October/000701.html
> 
> No, we are not reverting. Here's my current thought as a review:
> https://review.openstack.org/#/c/240371

Sigh, I was already worrying if this was going to break some tooling
somewhere.

Thanks, Dims for the quick response, and Matt for spotting it.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Change from Mitaka: Expected UNIX signal to generate Guru Meditation (error) Reports

[Kashyap Chamarthy]

Background
--

Oslo Guru Meditation (error) Reports (GMR)[*] are a useful debugging
mechanism that allows one to capture the current state of a Nova
process/executable (e.g. `nova-compute`, `nova-api`, etc).

The way to generate the error report is to supply the 'User-defined
signal', SIGUSR1, when killing a Nova process.  E.g.

$ kill -USR1 `pgrep nova-compute`

which results in GMR being printed to your standard error ('stderr')
stream, wherever it ends up being redirected to (e.g. to a corresponding
Nova process-specific log file, otherwise, on systemd-enabled systems,
to its journal).


Change in Mitaka (and above)


>From the upcoming Mitaka release onwards, the default expected UNIX
signal to generate GMR has been changed[1] from USR1 to USR2 (another
User-defined singal), because the USR1 is reserved by Apache 'mod_wsgi'
for its own purpose.

So, to generate GMR, from Mitaka release:

$ kill -USR2 `pgrep nova-compute`

A corresponding Nova documentation change[2] has been submitted to
reflect this new reality.


[1] https://review.openstack.org/#/c/223133/ -- guru_meditation_report:
Use SIGUSR2 instead of SIGUSR1 
[2] https://review.openstack.org/#/c/227779/ -- doc: gmr: Update
instructions to generate GMR error reports


[*] References
--

Related reading:

- http://docs.openstack.org/developer/nova/gmr.html
- http://docs.openstack.org/developer/oslo.reports/usage.html
- https://wiki.openstack.org/wiki/GuruMeditationReport
- 
https://www.berrange.com/posts/2015/02/19/nova-and-its-use-of-olso-incubator-guru-meditation-reports/

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Kashyap Chamarthy]

On Fri, Oct 02, 2015 at 06:20:31AM +, Koniszewski, Pawel wrote:
> > -Original Message-
> > From: Mathieu Gagné [mailto:mga...@internap.com]
> > Sent: Thursday, October 1, 2015 7:24 PM
> > To: OpenStack Development Mailing List (not for usage questions)

[. . .]

> > >> I have taken the liberty of listing those that responded to the
> > >> thread and the authors of mentioned patches as interested people.
> > >
> > >> From the responses and looking at the specs up for review it looks
> > >> like there are about five areas that could be addressed in Mitaka and
> > >> several others that could come later. The first five are:
> > >>
> > >>
> > >> - migrating instances with a mix of local disks and cinder volumes
> > >
> > > IIUC, this is possible with the selective block device migration work
> > > merged in upstream libvirt:
> > >
> > > https://www.redhat.com/archives/libvir-list/2015-May/msg00955.html
> > >
> > 
> > Can someone explain to me what is the actual "disk name" I have to pass in
> > to libvirt? I couldn't find any documentation about how to use this
> feature.
> 
> You have to pass device names from /dev/, e.g., if a VM has ephemeral disk
> attached at /dev/vdb you need to pass in 'vdb'. Format expected by
> migrate_disks is ",...".

Yeah, you can enumerate the current block devices for an instance by
doing:

$ virsh domblklist instance-0001

[If you're curious, the 'v' in the 'vda/vdb' stands for 'virtio' disks.
For non-virtio disks, you'd see a device name like 'hda'.]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Kashyap Chamarthy]

On Wed, Sep 30, 2015 at 11:25:12AM +, Murray, Paul (HP Cloud) wrote:
> 
> > Please respond to this post if you have an interest in this and what
> > you would like to see done.  Include anything you are already
> > getting on with so we get a clear picture. 
>
> Thank you to those who replied to this thread. I have used the
> contents to start an etherpad page here:
>
> https://etherpad.openstack.org/p/mitaka-live-migration

I added a couple of URLs for upstream libvirt work that allow for
selective block device migration, and the in-progress generic TLS
support work by Dan Berrange in upstream QEMU.

> I have taken the liberty of listing those that responded to the thread
> and the authors of mentioned patches as interested people.
 
> From the responses and looking at the specs up for review it looks
> like there are about five areas that could be addressed in Mitaka and
> several others that could come later. The first five are:
>
> 
> - migrating instances with a mix of local disks and cinder volumes 

IIUC, this is possible with the selective block device migration work
merged in upstream libvirt:

https://www.redhat.com/archives/libvir-list/2015-May/msg00955.html

> - pause instance during migration
> - cancel migration 
> - migrate suspended instances 
> - improve CI coverage
> 
> Not all of these are covered by specs yet and all the existing specs
> need reviews. Please look at the etherpad and see if there is anything
> you think is missing.
> 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][elections] PTL nomination period is now over

[Kashyap Chamarthy]

On Thu, Sep 17, 2015 at 05:09:04PM -0400, Nikhil Komawar wrote:

[. . .]

> > I think this is all superfluous however and we should simply encourage
> > people to not wait until the last minute. Waiting to see who is
> > running/what the field looks like isn't as important as standing up
> > and saying you're interested in running.
> 
> I like that you have used the word encourage however, will have to
> disagree here. Life in general can't permit that to everyone -- there
> can be any important things pop up at unexpected time, someone on
> vacation and getting late to come back etc. And on top of that people
> can get caught up particularly at this week.  The time-line for
> proposals is a good idea seems a good idea in general.

Morgan is absolutely right.

The risk of unexpected things cropping up is always there.  If one has
the _intention_ to run for PTL, then they should make it a priority to
send the nomination as early as possible once timelines are announced
(more so if you're an incumbent PTL), rather than waiting for the last
moment.

Also, don't miss the unambiguously clear comment from Matt Riedemann:

"If running for PTL is something you had in mind to begin with, you
should probably be looking forward to when the elections start and
get your ducks in a row.  Part of being PTL, a large part I'd think,
is the ability to organize and manage things. If you're waiting
until the 11th hour to do this, I wouldn't have much sympathy."

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] The unbearable lightness of specs

[Kashyap Chamarthy]

On Wed, Jun 24, 2015 at 02:51:38PM +0100, Nikola Đipanov wrote:
 On 06/24/2015 02:33 PM, Matt Riedemann wrote:

[. . .]

  I agree completely. The nicely rendered feature docs which is a
  byproduct of the specs process in gerrit is a great part of it. So when
  someone is trying to use a new feature or trying to fix a bug in said
  feature 1-2 years later and trying to understand the big picture idea,
  they can refer to the original design spec - assuming it was accurate at
  the time that the code was actually merged. Like you said, it's
  important to keep the specs up to date based on what was actually
  approved in the code.

 Of course documentation is good. Make that kind of docs a requirement
 for merging a feature, by all means.
 
 But the approval process we have now is just backwards. It's only result
 is preventing useful work getting done.
 
 In addition to what Daniel mentioned elsewhere:
 
 Why do cores need approved specs for example - and indeed for many of us
 - it's just a dance we do. I refuse to believe that a core can be
 trusted to approve patches but not to write any code other than a bugfix
 without a written document explaining themselves, and then have a yet
 more exclusive group of super cores approve that. It makes no sense.

This is one of the _baffling_ aspects -- that a so-called super core
has to approve specs with *no* obvious valid reasons.  As Jay Pipes
mentioned once, this indeed seems like a vestigial remnant from old
times.

FWIW, I agree with others on this thread, Nova should get rid of this
specific senseless non-process.  At least a couple of cycles ago.

[Snip, some sensible commentary.]


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] The unbearable lightness of specs

[Kashyap Chamarthy]

On Wed, Jun 24, 2015 at 04:09:16PM +0200, Kashyap Chamarthy wrote:
 On Wed, Jun 24, 2015 at 02:51:38PM +0100, Nikola Đipanov wrote:
  On 06/24/2015 02:33 PM, Matt Riedemann wrote:
 
 [. . .]
 
   I agree completely. The nicely rendered feature docs which is a
   byproduct of the specs process in gerrit is a great part of it. So when
   someone is trying to use a new feature or trying to fix a bug in said
   feature 1-2 years later and trying to understand the big picture idea,
   they can refer to the original design spec - assuming it was accurate at
   the time that the code was actually merged. Like you said, it's
   important to keep the specs up to date based on what was actually
   approved in the code.
 
  Of course documentation is good. Make that kind of docs a requirement
  for merging a feature, by all means.
  
  But the approval process we have now is just backwards. It's only result
  is preventing useful work getting done.
  
  In addition to what Daniel mentioned elsewhere:
  
  Why do cores need approved specs for example - and indeed for many of us
  - it's just a dance we do. 

On this part, I fully agree with Dan Smith's reasoning and from my own
observation of other communities I that follow (e.g. KVM/QEMU/libvirt),
every long time dev/reviewer  ('core' in OpenStack parlance)_does_
provide their intent/design thoughts in written form to the list to
discuss, iterate, and get different technical perspectives before going
ahead and implementing them.

  I refuse to believe that a core can be
  trusted to approve patches but not to write any code other than a bugfix
  without a written document explaining themselves, and then have a yet
  more exclusive group of super cores approve that. It makes no sense.
 
 This is one of the _baffling_ aspects -- that a so-called super core
 has to approve specs with *no* obvious valid reasons.  As Jay Pipes
 mentioned once, this indeed seems like a vestigial remnant from old
 times.

Just to clarify, that I only find it weird that there exists a special
sub-group for specs.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] The unbearable lightness of specs

[Kashyap Chamarthy]

On Wed, Jun 24, 2015 at 10:02:27AM -0500, Matt Riedemann wrote:
 
 
 On 6/24/2015 9:09 AM, Kashyap Chamarthy wrote:
 On Wed, Jun 24, 2015 at 02:51:38PM +0100, Nikola Đipanov wrote:
 On 06/24/2015 02:33 PM, Matt Riedemann wrote:

[. . .]

 This is one of the _baffling_ aspects -- that a so-called super core
 has to approve specs with *no* obvious valid reasons.  As Jay Pipes
 mentioned once, this indeed seems like a vestigial remnant from old
 times.
 
 FWIW, I agree with others on this thread, Nova should get rid of this
 specific senseless non-process.  At least a couple of cycles ago.
 
 Specs were only added a couple of cycles ago... :)  And they were added to
 fill a gap, which has already been pointed out in this thread.  So if we
 remove them without a replacement for that gap, we regress.

Oops, I didn't mean to say that Specs as a concept should be gone.
Sorr for poor phrasing.

My question was answred by Joe Gordon with this review:

https://review.openstack.org/#/c/184912/


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Follow up actions from the Summit: please help

[Kashyap Chamarthy]

On Fri, Jun 05, 2015 at 10:47:31AM +0100, John Garbutt wrote:
 Hi,
 
 So in the interests of filling up your inbox yet further...
 
 We have lots of etherpads from the summit:
 https://wiki.openstack.org/wiki/Design_Summit/Liberty/Etherpads#Nova
 
 I have extracted all the action items here:
 https://etherpad.openstack.org/p/YVR-nova-liberty-summit-action-items

Just a meta comment. Since the content is not massive, it'd be nice to
have it exported into plain textand post to the mailing list instead,
of being lost in a slow, etherpad of doom as you yourself put it on
IRC. :-)

A mailing list thread is faster to load than an etherpad.

[. . .]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Need volunteers for tempest bug triage

[Kashyap Chamarthy]

On Sat, May 30, 2015 at 03:52:02PM +0300, Yaroslav Lobankov wrote:
 Hi everyone,
 
 Is it possible for other people (not only core reviewers) to
 participate in bug triage? I would like to help in doing this.

Absolutely. There's no such silly rule that only core reviwers can do
bug triage.

While not mandatory, it can make your life a bit more easier while
troubleshooting if you have spent some time test/debugging OpenStack
environments. 

Take a look here:

https://wiki.openstack.org/wiki/BugTriage

Also, you might want to refer this useful notes from Sean Dague about
what to consider while triaging bugs (though, the example below is from
the Nova bug tracker, it's generally applicable across components):


http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Can we bump MIN_LIBVIRT_VERSION to 1.2.2 in Liberty?

[Kashyap Chamarthy]

On Thu, May 14, 2015 at 02:23:25PM -0500, Matt Riedemann wrote:
 The minimum required version of libvirt in the driver is 0.9.11 still [1].
 We've been gating against 1.2.2 in Ubuntu Trusty 14.04 since Juno.
 
 The libvirt distro support matrix is here: [2]
 
 Can we safely assume the people aren't going to be running Libvirt compute
 nodes on RHEL  7.1 or Ubuntu Precise?

I do come across occasional bugs where people were using mixed
environments with Compute nodes on RHEL6. But that doesn't mean,
OpenStack Gate should continue to use ancient versions.

 Regarding RHEL, I think this is a safe bet because in Kilo nova dropped
 python 2.6 support and RHEL  6 doesn't have py26 so you'd be in trouble
 running kilo+ nova on RHEL 6.x anyway.
 
 There are some workarounds in the code [4] I'd like to see removed by
 bumping the minimum required version.

It'd be helpful to see this change (increasing the min libvirt version)
in Gate, not least because it saves time spent on debugging issues that
were fixed in newer libvirt releases.

Speaking of versions, libvirt 1.2.2 was released on 02-MAR-2014 and QEMU
1.3.0 was released on 03-DEC-2012 -- both sound more than reasonable,
FWIW.

Maybe QEMU min version can be bumped too?

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-docker] Status update

[Kashyap Chamarthy]

On Sat, May 09, 2015 at 04:55:47PM +, Adrian Otto wrote:
 
 I will also mention that it’s natural to be allergic to the idea of
 nested virtualization. We all know that creating multiple levels of
 hardware virtualization leads to bad performance outcomes.

This seems to paint an overly bleak picture of nested virtualization.
Please allow me to explain.

Speaking from frequent upstream testing, KVM on KVM nested
virtualization, is far more performant[1][2] than nesting with plain
emulation (maybe you're referring to this). also stated by James
Bottomley elsewhere in this thread).

A small test (was done in OCT, 2013): 

A _single_ `make` process (10 iterations) to compile Linux Kernel,
with nested EPT[1] (Extended Page Tables) enabled. Perform the test
on a baremetal host (L0); a guest hypervisor (L1), running on L0;
and a nested guest (L2), running on L1.

Test script:

--
#!/bin/bash
cd /home/test/src/linux

make defconfig

for i in {1..10}; do
 make clean;
 sync;
 sudo sh -c echo 3  /proc/sys/vm/drop_caches
 time make;
done 21 | tee make-timings.txt
--

Result of time to `make defconfig`, average of 10 iterations on:

L2: 4m53s
L1: 3m56s
L0: 2m14s

As you can see from the timings, for some kind of workloads (e.g. CPU
intensive) KVM nested virt is not as bad.  And, it's worth noting that
this test was done without any other performance tuning. 

Of course, this doesn't mean that it's suitable for all use cases or
there's no complexity. This needs more testing and providing feedback to
upstream KVM/Kernel community. FWIW, in my experience of reporting bugs
in this area, upstream KVM maintainers have been very responsive.


[1] Talk on the said subject by Gleb Natapov, KVM maintainer in 2013 --
http://www.linux-kvm.org/images/8/8c/Kvm-forum-2013-nested-ept.pdf
[2] Talk by Jan Kizka, Bandan Das et al. at 2014 KVM Forum, with details on
performance evaluation --
http://www.linux-kvm.org/images/3/33/02x03-NestedVirtualization.pdf

-- 
/kashyap

 However, nested containers do not carry that same drawback, because
 the actual overhead of a Linux cgroup and Kernel Namespeaces are much
 lighter than a hardware virtualization. There are cases where having a
 container-in-container setup gives compelling benefits. That’s why
 I’ll argue vigorously for both Nova and Magnum to be able to produce
 container instances both at the machine level, and allow Magnum to
 produce nested containers” to produce better workload consolidation
 density. in a setup with no hypervisors at all.
 
 To sum up, I strongly support merging in nova-docker, with the caveat
 that it operates within the existing Nova API (with few minor
 exceptions). For features that require API features that are truly
 container specific, we should land those in Magnum, and keep the Nova
 API scoped to operations that are appropriate for “all instance
 types.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] cross project communication: periodic developer newsletter?

[Kashyap Chamarthy]

On Mon, May 04, 2015 at 07:22:24PM +, Nikhil Komawar wrote:
 
 This is a really nice idea. 

Idea is indeed nice, but I have to fully agree with all of what Theirry
says.

 I feel the same, that we can offload some

I personally don't feel it's about offloading to some hypothetical
person who'll magically understand (_and_ keep up) the detailed
technical context across the projects to produce such a high-quality
developer-oriented newsletter.

Taking the venerable LWN as example, those who write most of the
articles are folks who're technically very involved (not at a hand-wavy
level, giving some directions) in Kernel and related projects. The
thorough research completely shines through. It's more than a full-time
job given the number of OpenStack projects, and assuming the the cadence
of such a newsletter is once a month.

 of the work from the liaisons and reduce the number of syncs that need
 to happen. This can be a good source of asynchronous communication
 however, I still feel that we need to keep a good balance of both.
 Also, I like the proposed scope of the newsletter.

[. . .]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

[Kashyap Chamarthy]

On Tue, Apr 21, 2015 at 02:58:57PM -0700, Joe Gordon wrote:
 On Fri, Apr 17, 2015 at 1:22 AM, Victor Stinner vstin...@redhat.com wrote:
 
   For the full list, see the wiki page:
   https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects
 
   Thanks for updating the wiki page that is a very useful list.
   From the looks of things, it seems like nova getting Python3 support in
  Liberty is not going to happen.
 
 
 
 I based this on the wiki, but maybe I am wrong.
 
 remaining libraries for nova:
 oslo.db -- looks like it is almost there
 oslo.messaging  -- same
 paste -- almost there
 sqlalchemy-migrate -- almost there
 suds -- with the suds fork shouldn't be too hard
 websockify -- unknown


 libvirt-python -- unknown

I notice some work from Dan Berrange in late 2013 about this:

  https://www.redhat.com/archives/libvir-list/2013-December/msg00171.html
  -- [PATCH python 00/15] Initial work porting to python3

And, I see several commits in libvirt-python git repo in related to
Python 3.

Maybe Dan can comment when he notices this.

 mysql-python  -- alternitive looks viable.
 
 Based on there being two unknowns, and a lot of dependencies that are just
 almost there, and a few we may want to migrate off of, I was assuming
 addressing those issues would make it hard for us to make nova python3
 compatible for Liberty.
 

[. . .]

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] VM migration between two data centers

[Kashyap Chamarthy]

On Fri, Apr 17, 2015 at 04:05:20PM +0530, Abhishek Talwar/HYD/TCS wrote:
 Hi Folks,
 
 I have created two data centers and I am using OpenStack as the
 management platform for them. So now my question is it is possible to
 migrate VM instances from one data center to the other. 

Please ask such questions on http://ask.openstack.org/.  Though you need
to rephrase your question with more concrete details.

This mailing list of for development discussions.

 Can two OpenStack clouds migrate VM to each other ?

They can -- look up the documentation available around this.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Regarding deleting snapshot when instance is OFF

[Kashyap Chamarthy]

On Wed, Apr 08, 2015 at 11:31:40PM +0530, Deepak Shetty wrote:
 Hi,
 Cinder w/ GlusterFS backend is hitting the below error as part of
 test_volume_boot_pattern tempest testcase

[Meta comment: Since main components that are triggering this errors are
Cinder with GlusterFS, adding Cinder tag would be useful to raise the
right folks' attention.]

 (at the end of testcase when it deletes the snap)
 
 /usr/local/
 
 lib/python2.7/dist-packages/libvirt.py, line 792, in blockRebase
 2015-04-08 07:22:44.376 32701 TRACE nova.virt.libvirt.driver if ret == -1:
 raise libvirtError ('virDomainBlockRebase() failed', dom=self)
 2015-04-08 07:22:44.376 32701 TRACE nova.virt.libvirt.driver
 libvirtError: *Requested
 operation is not valid: domain is not running*
 2015-04-08 07:22:44.376 32701 TRACE nova.virt.libvirt.driver

You'll likely see more details in libvirt's log why virDomainBlockRebase
fails. If you hit this failure on any of the recent Gate runs, then the
libvirt debug logs (now enabled by default) might give some clue.

Also, it would be useful if you can reproduce this issue outside of
Tempest (and its timing issues). Even better, if you can reproduce this
failure w/ just plain Cinder (or even w/o Cinder) to isolate the issue.

 More details in the LP bug [1]

The details in the bug does not provide a reproducer. As always,
providing a crystal clear reproducer (e.g. a script, or sequence of
`virsh`/libvirt API calls or exact Nova/Cinder commands) leading to the
failure will allow people to take a look at the bug much more quicker
Instead of leaving the Burden of Proof on the bug triagers to have a
reproducer.

 In looking closely at the testcase, it waits for the Instance to turn
 OFF post which the cleanup starts which tried to delete the snap, but
 since the cinder volume is attached state (in-use) it lets nova take
 control of the snap del operation, and nova fails as it cannot do
 blockRebase as domain is offline.


blockRebase (in short, it populates a disk image with data from its
:backing image chain, and can act on different flags you provide to it)
cannot operate on an offline image (nor on a persistent libvirt domain,
but Nova deals with it by temporarily undefining it and later redefining
it). So, first you might want to figure out why the guest is offline
before blockRebase call is invoked to get an understanding of your
questions below.

 Questions:
 
 1) Is this a valid scenario being tested ? Some say yes, I am not
 sure, since the test makes sure that instance is OFF before snap is
 deleted and this doesn't work for fs-backed drivers as they use hyp
 assisted snap which needs domain to be active.
 
 
 2) If this is valid scenario, then it means libvirt.py in nova should
 be modified NOT to raise error, but continue with the snap delete (as
 if volume was not attached) and take care of the dom xml (so that
 domain is still bootable post snap deletion), is this the way to go ?
 
 
 Appreciate suggestions/comments


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug expiration

[Kashyap Chamarthy]

On Thu, Apr 02, 2015 at 11:32:44AM +0200, Thierry Carrez wrote:
 Sean Dague wrote:
  I just spent a chunk of the morning purging out some really old
  Incomplete bugs because about 9 months ago we disabled the auto
  expiration bit in launchpad -
  https://bugs.launchpad.net/nova/+configure-bugtracker
  
  This is a manually grueling task, which by looking at these bugs, no one
  else is doing. I'd like to turn that bit back on so we can actually get
  attention focused on actionable bugs.
  
  Any objections here?
 
 No objection, just a remark:
 
 One issue with auto-expiration is that it usually results in the
 following story:
 
 1. Someone reports bug
 2. Triager notices NEW bug, asks reporter for details, sets INCOMPLETE
 3. Reporter provides details
 4. Triager does not notice reply on bug since they ignore INCOMPLETE
 5. Bug expires after n months and disappears forever
 6. Reporter is frustrated and won't ever submit issues again
 
 The problem is of course at step 4, not at step 5. Auto-expiration is
 very beneficial if your bug triaging routine includes checking Launchpad
 for INCOMPLETE bugs with an answer regularly. If nobody does this very
 boring task, then auto-expiration can be detrimental.

This is an excellent point. 

A reporter takes time to file a bug, it should not be mindlessly expired
via a bot (as Invalid rationale) without even any priliminary
investigation in the first place.

Unfortunately, I see most attention is only on refactoring or on
half-baked shiny new features, which fall apart if you sneeze. Perils of
moving fast.

 Is anyone in Nova checking for INCOMPLETE bugs with an answer ? That's
 task 4 in https://wiki.openstack.org/wiki/BugTriage ...

I doubt that, I only see Sean take up this drudgery most times. I should
admit, I try sometimes, but just feel overwhelmed with the sheer numbers
and get distracted with other work.

PS: On a side note, I'd wish Launchpad (or the upcoming Storyboard) has
a state like INSUFFICIENT_DATA instead of the current, lousy,
Invalid state (using it as a blanket rationale for most bugs we want
to close). A bug can be _valid_ but might not have sufficient data for
any no. of reasons, it ought to be closed, accurately, as
INSUFFICIENT_DATA not Invalid.


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack]Specific Juno version

[Kashyap Chamarthy]

On Wed, Mar 04, 2015 at 02:41:53PM +0200, Eduard Matei wrote:
 Hi Ihar,
 This is the error i see in c-vol screen:
 2015-03-03 18:39:34.060 DEBUG taskflow.engines.action_engine.runner
 [req-87db2771-5c25-4c9b-a3bc-f7a504468315 4af0a30fe2c14deb9f9cf341a97daa9f
 1fbab1e235ff414abf249c741fb3c6c9] Exiting old state 'WAITING' in response
 to event 'analyze' from (pid=29710) on_exit
 /usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:156
 2015-03-03 18:39:34.060 DEBUG taskflow.engines.action_engine.runner
 [req-87db2771-5c25-4c9b-a3bc-f7a504468315 4af0a30fe2c14deb9f9cf341a97daa9f
 1fbab1e235ff414abf249c741fb3c6c9] Entering new state 'ANALYZING' in
 response to event 'analyze' from (pid=29710) on_enter
 /usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/runner.py:160
 2015-03-03 18:39:34.061 CRITICAL cinder [-] TypeError: an integer is
 required
 
 2015-03-03 18:39:34.061 TRACE cinder Traceback (most recent call last):
 2015-03-03 18:39:34.061 TRACE cinder   File
 /opt/stack/devstack/cinder/bin/cinder-volume, line 79, in module
 2015-03-03 18:39:34.061 TRACE cinder launcher.wait()
 2015-03-03 18:39:34.061 TRACE cinder   File
 /opt/stack/devstack/cinder/cinder/openstack/common/service.py, line 393,
 in wait
 2015-03-03 18:39:34.061 TRACE cinder self._respawn_children()
 2015-03-03 18:39:34.061 TRACE cinder   File
 /opt/stack/devstack/cinder/cinder/openstack/common/service.py, line 381,
 in _respawn_children
 2015-03-03 18:39:34.061 TRACE cinder self._start_child(wrap)
 2015-03-03 18:39:34.061 TRACE cinder   File
 /opt/stack/devstack/cinder/cinder/openstack/common/service.py, line 327,
 in _start_child
 2015-03-03 18:39:34.061 TRACE cinder os._exit(status)
 2015-03-03 18:39:34.061 TRACE cinder TypeError: an integer is required
 2015-03-03 18:39:34.061 TRACE cinder
 2015-03-03 18:39:34.375 INFO cinder.openstack.common.service [-] Child
 29710 exited with status 1
 2015-03-03 18:39:34.378 INFO cinder.openstack.common.service [-] Started
 child 29948
 2015-03-03 18:39:34.380 INFO cinder.service [-] Starting cinder-volume node
 (version 2014.2.3)


I very recently tested stable Juno DevStack (which in turn, checks out
respective stable branches of other projects) successfully, (but w/o
Cinder, I should mention):

$ git checkout remotes/origin/stable/juno

With these services enabled:


ENABLED_SERVICES=g-api,g-reg,key,n-api,n-cpu,n-sch,n-cond,mysql,rabbit,dstat,quantum,q-svc,q-agt,q-dhcp,q-l3,q-meta


 Regarding the disclaimer, that's the standard signature we have to use in
 the company.

Could be. But you cannot enforce such legalese on a public mailing list
-- that's what Ihar was trying to say. Thanks for correcting that in
your next response.

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Kashyap Chamarthy]

On Tue, Feb 24, 2015 at 10:02:36PM -0800, Mark Atwood wrote:
 On Tue, Feb 24, 2015, at 04:28, Kashyap Chamarthy wrote:
  
  Along with the below, if push comes to shove, OpenStack Foundation could
  probably try a milder variant (obviously, not all activities can be
  categorized as 'critical path') of Linux Foundation's Critical
  Infrastructure Protection Initiative[1] to fund certain project
  activities in need.
 
 Speaking as a person who sits on the LF CII board meetings,
 and helps turn the crank on that particular sausage mill,
 no, we really don't want to go down that path at this point in
 time.

I didn't imply to do an _exact_ version of LF's CII. Given so much of
interest in OpenStack, vendors/interested parties must (yes) maintain a
balance between giving vs taking from the community. And, they should be
provided ample information on _where_ they can target people/engineers
(Dan Berrange noted this too in one of his responses in this thread).

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Kashyap Chamarthy]

On Tue, Feb 24, 2015 at 11:54:31AM +, Daniel P. Berrange wrote:
 On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
  On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
  
  need to do more work. If this is so, then I don't think this is a blocker,
  it is just a sign that the project needs to focus on providing more 
  resources
  to the teams impacted in that way.
  
  What are the mechanisms whereby the project provides more resources
  to teams?

Along with the below, if push comes to shove, OpenStack Foundation could
probably try a milder variant (obviously, not all activities can be
categorized as 'critical path') of Linux Foundation's Critical
Infrastructure Protection Initiative[1] to fund certain project
activities in need.
 
 The technical committee and / or foundation board can highlight the
 need for investment of resources in critical areas of the project, to
 either the community members or vendors involved. As an example, this
 was done successfully recently to increase involvement in maintaining
 the EC2 API support.  There are plenty of vendors involved in
 OpenStack which have the ability to target resources, if they can
 learn where those resources are best spent.
 

[1] http://www.linuxfoundation.org/programs/core-infrastructure-initiative

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] authentication issues

[Kashyap Chamarthy]

On Mon, Feb 16, 2015 at 09:54:30AM +, Gary Kotton wrote:
 Hi, With the current master in devstack I am unable to create a
 neutron network. 

 It looks like there is an issue with keystone. Anyone
 else hit this?  Thanks Gary

Hmm, I don't see any such issue here, I'm using:
Nova+Neutron+Keystone+Glance with libvirt/KVM drivers.

And, these are the commits I'm at:

  DevStack
  commit 13c7ccc9d5d7ee8b88c2ee7d4af8990a075440a2
  Glance
  commit cd60a24a7d32d4ca0be36f7afa4d082193958989
  Merge: 803c540 4a78e85
  Keystone
  commit c06591dd470aaa595206100d0176ccc0575d58b7
  Author: OpenStack Proposal Bot openstack-in...@lists.openstack.org
  Neutron
  commit 12b0d4d9792d83fd559de59f2dd7b9f532f89398
  Merge: 7a0a2a1 a3ab3eb
  Nova
  commit 69f4b44691ddabc2cfa8c08539a51d255646e173
  Merge: ea1f1d6 353e823
  requirements
  commit ec1c788c2bfac3ef964396b92f8a090b60dbb4ef

$ neutron net-list
+--+-++
| id   | name| subnets  
  |
+--+-++
| ec39ff23-9319-4526-aa55-fdd0ec172bc5 | private | 
2192c1b9-d1de-406d-ab7e-ba142a9b7d3d 10.1.0.0/24   |
| d2994e9c-649a-4668-a6bc-a71ff2362e74 | public  | 
94cf7c03-7352-4b8d-8b00-21227c7414a4 172.24.4.0/24 |
+--+-++

$ ps -ef | grep -i neutron
kashyapc 12983 12954  1 05:43 pts/700:00:03 python /usr/bin/neutron-server 
--config-file /etc/neutron/neutron.conf --config-file 
/etc/neutron/plugins/ml2/ml2_conf.ini
kashyapc 13040 13014  0 05:43 pts/800:00:02 python 
/usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf 
--config-file /etc/neutron/plugins/ml2/ml2_conf.ini
kashyapc 13159 13049  0 05:43 pts/900:00:00 python 
/usr/bin/neutron-dhcp-agent --config-file /etc/neutron/neutron.conf 
--config-file=/etc/neutron/dhcp_agent.ini
root 13252 13040  0 05:43 pts/800:00:00 sudo /usr/bin/neutron-rootwrap 
/etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport 
--format=json
root 13254 13252  0 05:43 pts/800:00:00 /usr/bin/python 
/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor 
Interface name,ofport --format=json
kashyapc 13263 13175  0 05:43 pts/10   00:00:00 python 
/usr/bin/neutron-l3-agent --config-file /etc/neutron/neutron.conf 
--config-file=/etc/neutron/l3_agent.ini
kashyapc 13348 13273  0 05:43 pts/11   00:00:00 python 
/usr/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf 
--config-file=/etc/neutron/metadata_agent.ini
kashyapc 13369 13348  0 05:43 pts/11   00:00:00 python 
/usr/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf 
--config-file=/etc/neutron/metadata_agent.ini
kashyapc 13370 13348  0 05:43 pts/11   00:00:00 python 
/usr/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf 
--config-file=/etc/neutron/metadata_agent.ini
kashyapc 13371 13348  0 05:43 pts/11   00:00:00 python 
/usr/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf 
--config-file=/etc/neutron/metadata_agent.ini
kashyapc 13372 13348  0 05:43 pts/11   00:00:00 python 
/usr/bin/neutron-metadata-agent --config-file /etc/neutron/neutron.conf 
--config-file=/etc/neutron/metadata_agent.ini
kashyapc 13940 1  0 05:43 ?00:00:00 /usr/bin/python 
/usr/bin/neutron-ns-metadata-proxy 
--pid_file=/home/kashyapc/src/cloud/data/neutron/external/pids/ef5ae22b-86d2-42c0-8be4-95f75683bbfd.pid
 --metadata_proxy_socket=/home/kashyapc/src/cloud/data/neutron/metadata_proxy 
--router_id=ef5ae22b-86d2-42c0-8be4-95f75683bbfd 
--state_path=/home/kashyapc/src/cloud/data/neutron --metadata_port=9697 
--metadata_proxy_user=1000 --metadata_proxy_group=1000 --debug --verbose



-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Flavor extra-spec and image metadata documentation

[Kashyap Chamarthy]

On Wed, Feb 11, 2015 at 10:23:54AM +0100, Pasquale Porreca wrote:
 Hello
 
 I am working on a little patch that introduce a new flavor extra-spec
 and image metadata key-value pair https://review.openstack.org/#/c/153607/
 
 I am wondering how an openstack admin can be aware that a specific value
 of a flavor extra-spec or image metadata provides a feature he may
 desire, in other words is there a place where the flavor extra-specs
 and/or image metadata key-value pairs are documented?

Unfortunately, there's none as of now. I found this the hard way that
you cannot trivially find all possible 'extra_spec' key values that can
be set by `nova flavor-key`

I did gross things like this:

$ grep hw\: nova/virt/hardware.py nova/tests/unit/virt/test_hardware.py | 
sort | uniq

And, obviously the above will only find you 'hw' properties.

Daniel Berrangé once suggested that image properties and flavor extra
specs need to be 'objectified' to alleviate this.

 I found plenty of documentation on how to list, create, delete, etc.
 flavor extra-spec and image metadata, but the only place where I found a
 list (is that complete?) of the accepted (i.e. that trigger specific
 feature in nova) key-value pairs is in horizon dashboard, when logged
 with admin credential.
 
 I am a bit confused on how someone working to add a new key/value pair
 should proceed. 
 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] Lets keep our community open, lets fight for it

[Kashyap Chamarthy]

On Wed, Feb 11, 2015 at 12:53:24PM +0100, Nikola Đipanov wrote:
 + Inf for writing this Flavio!

Absolutely! I never even knew such things existed in this community.

 Only some observations below.
 
[. . .]

  ## Mailing List vs IRC Channel
  
  I get it, our mailing list is freaking busy, keeping up with it is
  hard and time consuming and that leads to lots of IRC discussions. I
  don't think there's anything wrong with that but I believe it's wrong
  to expect *EVERYONE* to be in the IRC channel when those discussions
  happen.
  
  If you are discussing something on IRC that requires the attention of
  most of your project's community, I highly recommend you to use the
  mailing list as oppose to pinging everyone independently and fighting
  with time zones. Using IRC bouncers as a replacement for something
  that should go to the mailing list is absurd. Please, use the mailing
  list and don't be afraid of having a bigger community chiming in in
  your discussion.  *THAT'S A GOOD THING*
  
  Changes, specs, APIs, etc. Everything is good for the mailing list.
  We've fought hard to make this community grow, why shouldn't we take
  advantage of it?
  
 
 I think the above 2 are somewhat intertwined with another trend in the
 community I've personally noticed towards the end of the Juno cycle,
 that I also strongly believe needs to DIAFF.
 
 An idea that it is possible to manage and open source community using
 similar methods that are commonly used for managing subordinates in a
 corporate hierarchy.

That intention/notion to let's manage the community like a team just as
we do at good old $company should be absolutely demolished! People who
advocate such behavior are using ridiculously outdated brain models and
should drop what they're doing immediately and do some introspection.


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova

[Kashyap Chamarthy]

On Wed, Feb 04, 2015 at 11:15:18AM -0500, Davanum Srinivas wrote:
 Daniel, Kashyap,
 
 One question that came up on IRC was, how/where to configure say a
 directory where core dumps from qemu would end up. Sean was seeing a
 scenario where he noticed a core dump from qemu in dmesg/syslog 

If you're using something similar to ABRT on Ubuntu (assuming you're
using that), you should be able to find the stack traces. Seems like on
Ubuntu, this is what they use

https://wiki.ubuntu.com/Apport

 and  was wondering how to specify a directory to capture a core dump 
 if/when it occurs.

Incidentally, last week I was debugging a QEMU seg fault with upstream
folks[1] which involves qemu-nbd.

To capture coredumps with QEMU, this is what I had to do on my Fedora 21
system. Ensure abrt-ccpp (Automatic bug reporting tool is running:

  $ systemctl status abrt-ccpp
  ● abrt-ccpp.service - Install ABRT coredump hook
 Loaded: loaded (/usr/lib/systemd/system/abrt-ccpp.service; enabled)
 Active: active (exited) since Tue 2015-02-03 16:16:14 CET; 1 day 3h ago
   Main PID: 1113 (code=exited, status=0/SUCCESS)
 CGroup: /system.slice/abrt-ccpp.service

Then the coredump file should land in:

/var/tmp/abrt/ccpp-2015-01-30-00\:01\:05-3145/

You can actually see that the specific coredump is for QEMU, by running
(note, in this example it was `qemu-img` that was seg faulting - and
there are  patches accepted for this upstream):

$ cd  /var/tmp/abrt/ccpp-2015-01-30-00\:01\:05-3145/
$ file coredump 
coredump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, 
from '/home/kashyapc/build/qemu/qemu-img 


Assuming you have all the QEMU debug info packages installed, you can
then invoke GDB to capture the trace backs. An example[2]

  $ gdb /var/tmp/abrt/ccpp-2015-01-30-00\:01\:05-3145/coredump
  [. . .]
  (gdb) bt full
  [. . .]


  [1]
  http://lists.nongnu.org/archive/html/qemu-devel/2015-01/msg04397.html
  [2]
  
https://kashyapc.fedorapeople.org/virt/qemu-nbd-test/stack-traces-from-coredump.txt

 On Wed, Feb 4, 2015 at 8:48 AM, Kashyap Chamarthy kcham...@redhat.com wrote:
  On Wed, Feb 04, 2015 at 10:27:34AM +, Daniel P. Berrange wrote:
  On Wed, Feb 04, 2015 at 11:23:34AM +0100, Kashyap Chamarthy wrote:
   Heya,
  
   I noticed a ping (but couldn't respond in time) on #openstack-nova IRC
   about turning on logging in libvirt to capture Nova failures.
  
   This was discussed on this list previously by Daniel Berrange, just
   spelling it out here for reference and completness' sake.
  
  
   (1) To see the interactions between libvirt and QEMU, in
   /etc/libvirt/libvirtd.conf, have these two config attributes:
  
   . . .
   log_filters=1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 
   3:file 1:util
 
  You really want  3:object in there /before/  1:util too otherwise you'll
  be spammed with object ref/unref messages
 
  Thanks for this detail.
 
 
   log_outputs=1:file:/var/tmp/libvirtd.log
 
  Use  /var/log/libvirt/libvirtd.log instead of /var/tmp
 
  Ah, yeah, it was an incorrect copy/paste.
 
  --
  /kashyap
 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][libvirt] Logging interactions of libvirt + QEMU in Nova

[Kashyap Chamarthy]

Heya,

I noticed a ping (but couldn't respond in time) on #openstack-nova IRC
about turning on logging in libvirt to capture Nova failures.

This was discussed on this list previously by Daniel Berrange, just
spelling it out here for reference and completness' sake.


(1) To see the interactions between libvirt and QEMU, in
/etc/libvirt/libvirtd.conf, have these two config attributes:

. . .
log_filters=1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 
1:util
log_outputs=1:file:/var/tmp/libvirtd.log
. . .

(2) Restart libvirtd:

$ systemctl restart libvirtd

(3) Enable Nova debug logs in /etc/nova/nova.conf:

. . .  
verbose = True
debug = True
. . .

(4) Restart the Nova service:

$ openstack-service restart nova

(5) Repeat the offending test.

The libvirtd.log file should have the relevant details.


Other resources
---

- http://wiki.libvirt.org/page/DebugLogs
- https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] required libvirtd/qemu versions for numa support?

[Kashyap Chamarthy]

On Mon, Jan 26, 2015 at 03:37:48PM -0800, Jay Pipes wrote:
 On 01/26/2015 07:33 AM, Chris Friesen wrote:
 Hi,
 
 I'm interested in the recent work around NUMA support for guest
 instances
 (https://blueprints.launchpad.net/nova/+spec/virt-driver-numa-placement), but
 I'm having some difficulty figuring out what versions of libvirt and
 qemu are required.
 
  From the research that I've done it seems like qemu 2.1 might be
 required, but I've been unable to find a specific version listed in the
 nova requirements or in the openstack global requirements.  Is it there
 and I just can't find it?

Although, the MIN_LIBVIRT_NUMA_TOPOLOGY_VERSION points to libvirt 1.0.4. I
think newer the QEMU/libvirt, the better off you'll be. Specifically, upstream
libvirt has had some fixes in NUMA placement/vCPU pinning area -- so maybe you
can pick the newest upstream release 1.2.12, that was done today. (If you use
Fedora, it's already available in Fedora Rawhide distribution.)

 If it's not specified, and yet openstack relies on it, perhaps it should
 be added.  (Or at least documented somewhere.)
 
 Hi Chris,
 
 The constants starting here:
 
 http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n340
 
 should answer your questions.
 
 All the best,
 -jay
 

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][devstack][keystone] Devstack failures due to empty service catalogue

[Kashyap Chamarthy]

On Thu, Jan 22, 2015 at 09:42:50AM +0100, Jakub Libosvar wrote:
 It is caused by using old python-openstackclient [1]. We should be using
 = 1.0.2 - fix is about to be merged [2].

Thanks Jakub for helping me debug this on the IRC yesterday!

I hope these kind of backward incompatible changes can be avoided in
`openstack` client. It's really annoying to guess at these kind of
changes that consumes time for debugging (needlessly).


-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Using DevStack for multi-node setup

[Kashyap Chamarthy]

On Mon, Jan 05, 2015 at 08:20:48AM -0500, Sean Dague wrote:
 On 01/03/2015 04:41 PM, Danny Choi (dannchoi) wrote:
  Hi,
  
  I’m using DevStack to deploy OpenStack on a multi-node setup:
  Controller, Network, Compute as 3 separate nodes
  
  Since the Controller node is stacked first, during which the Network
  node is not yet ready, it fails to create the router instance and the
  public network.
  Both have to be created manually.
  
  Is this the expected behavior?  Is there a workaround to have DevStack
  create them?
 
 The only way folks tend to run multinode devstack is Controller +
 Compute nodes. And that sequence of creating an all in one controller,
 plus additional compute nodes later, works.

Sean, I wonder if you have a pointer to an example CI gate job (assuming
there's one) for the above with Neutron networking?


-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] 'module' object has no attribute 'HVSpec'

[Kashyap Chamarthy]

On Mon, Dec 22, 2014 at 04:37:47PM +0530, Srinivasa Rao Ragolu wrote:
 Hi All,
 
 I have integrated below CPU pinning patches to Nova

As of now, CPU pinning works directly from Nova git (as you can see,
most of the patches below are merged), you don't have to manually apply
any patches.

 https://review.openstack.org/#/c/132001/2https://review.openstack.org/#/c/128738/12https://review.openstack.org/#/c/129266/11https://review.openstack.org/#/c/129326/11https://review.openstack.org/#/c/129603/10https://review.openstack.org/#/c/129626/11https://review.openstack.org/#/c/130490/11https://review.openstack.org/#/c/130491/11https://review.openstack.org/#/c/130598/10https://review.openstack.org/#/c/131069/9https://review.openstack.org/#/c/131210/8https://review.openstack.org/#/c/131830/5https://review.openstack.org/#/c/131831/6https://review.openstack.org/#/c/131070/https://review.openstack.org/#/c/132086/https://review.openstack.org/#/c/132295/https://review.openstack.org/#/c/132296/https://review.openstack.org/#/c/132297/https://review.openstack.org/#/c/132557/https://review.openstack.org/#/c/132655/

The links are all mangled due to the bad formatting.

 And now if I try to run nova-compute, getting below error
 
 
 File /opt/stack/nova/nova/objects/compute_node.py, line 93, in 
 _from_db_object
 
 for hv_spec in hv_specs]
 
 AttributeError: 'module' object has no attribute 'HVSpec'

You can try directly from git and DevStack without applying manually
patches.

Also, these kind of usage questions are better suited for operator list
or ask.openstack.org.

-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug 1334398 and libvirt live snapshot support

[Kashyap Chamarthy]

On Mon, Dec 08, 2014 at 09:12:24PM +, Jeremy Stanley wrote:
 On 2014-12-08 11:45:36 +0100 (+0100), Kashyap Chamarthy wrote:
  As Dan Berrangé noted, it's nearly impossible to reproduce this issue
  independently outside of OpenStack Gating environment. I brought this up
  at the recently concluded KVM Forum earlier this October. To debug this
  any further, one of the QEMU block layer developers asked if we can get
  QEMU instance running on Gate run under `gdb` (IIRC, danpb suggested
  this too, previously) to get further tracing details.
 
 We document thoroughly how to reproduce the environments we use for
 testing OpenStack. 

Yep, documentation is appreciated.

 There's nothing rarified about a Gate run that anyone with access to
 a public cloud provider would be unable to reproduce, save being able
 to run it over and over enough times to expose less frequent failures.

Sure. To be fair, this was actually tried. At the risk of over
discussing the topic, allow me to provide a bit more context, quoting
Dan's email from an old thread[1] (Thoughts on the patch test failure
rate and moving forward Jul 23, 2014) here for convenience:

In some of the harder gate bugs I've looked at (especially the
infamous 'live snapshot' timeout bug), it has been damn hard to
actually figure out what's wrong. AFAIK, no one has ever been able
to reproduce it outside of the gate infrastructure. I've even gone
as far as setting up identical Ubuntu VMs to the ones used in the
gate on a local cloud, and running the tempest tests multiple times,
but still can't reproduce what happens on the gate machines
themselves :-( As such we're relying on code inspection and the
collected log messages to try and figure out what might be wrong.

The gate collects alot of info and publishes it, but in this case I
have found the published logs to be insufficient - I needed to get
the more verbose libvirtd.log file. devstack has the ability to turn
this on via an environment variable, but it is disabled by default
because it would add 3% to the total size of logs collected per gate
job.

There's no way for me to get that environment variable for devstack
turned on for a specific review I want to test with. In the end I
uploaded a change to nova which abused rootwrap to elevate
privileges, install extra deb packages, reconfigure libvirtd logging
and restart the libvirtd daemon.

   
https://review.openstack.org/#/c/103066/11/etc/nova/rootwrap.d/compute.filters
   https://review.openstack.org/#/c/103066/11/nova/virt/libvirt/driver.py

My next attack is to build a custom QEMU binary and hack nova
further so that it can download my custom QEMU binary from a website
onto the gate machine and run the test with it. Failing that I'm
going to be hacking things to try to attach to QEMU in the gate with
GDB and get stack traces.  Anything is doable thanks to rootwrap
giving us a way to elevate privileges from Nova, but it is a
somewhat tedious approach.


   [1] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041148.html

To add to the above, from the bug, you can find in one of the plenty of
invocations, the above issue _was_ reproduced once, albiet with
questionable likelihood (details in the bug).

So, it's not that what you're suggesting was never tried. But, from the
above, you can clearly see what kind of convoluted methods you need to
resort to.

One concrete point from the above: it'd be very useful to have an env
variable that can be toggled to enable libvirt/QEMU run under `gdb` for
$REVIEW.

(Sure, it's a patch that needs to be worked on. . .)

[. . .]

 The QA team tries very hard to make our integration testing
 environment as closely as possible mimic real-world deployment
 configurations. If these sorts of bugs emerge more often because of,
 for example, resource constraints in the test environment then it
 should be entirely likely they'd also be seen in production with the
 same frequency if run on similarly constrained equipment. And as we've
 observed in the past, any code path we stop testing quickly
 accumulates new bugs that go unnoticed until they impact someone's
 production environment at 3am.

I realize you're raising the point that it should not be taken lightly
-- hope the context provided in this email demonstrates that it's not
the case.


PS: FWIW, I do enable this codepath in my test environments (sure, it's
not *representative*), but I'm yet to reproduce the bug.


-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Scale out bug-triage by making it easier for people to contribute

[Kashyap Chamarthy]

On Tue, Nov 18, 2014 at 11:42:19AM +0100, Thierry Carrez wrote:
 Dolph Mathews wrote:
  As someone who has spent quite a bit of time triaging bugs, I'd be
  hugely in favor of this. I'd probably be willing to pitch in on
  additional projects, as well.
  
  Is there already tooling for this built around Launchpad, or do we have
  to roll our own?

FWIW, I find the CLI querying in Launchpad a bit lacking compared to
Bugzilla.
A while ago, Thierry pointed me to 'lp-tools'[1] which has a bunch of
CLI scripts to interface with Launchpad

I use it sometimes to check new Nova bugs:

$ python new-nova-lp-bugs.py | tee new-nova-lp-bugs-18NOV2014.txt

Couple of other resources I try to use[2][3] when I triage bugs:

[1] https://launchpad.net/lptools
[2] Notes from Sean Dague from Nova bug triaging --

http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html
[3] https://wiki.openstack.org/wiki/BugFilingRecommendations

 You'll have to roll your own, but it shouldn't be very difficult. You
 can see [1] for a base example of listing bugs in Launchpad:
 
 [1]
 http://git.openstack.org/cgit/openstack-infra/release-tools/tree/process_bugs.py
 
  With storyboard.openstack.org http://storyboard.openstack.org looming
  on the horizon, should such an effort be put on the back burner for now?
 
 I don't expect integrated projects to be able to switch to storyboard
 for bug tracking just yet. Our current goal is to have it ready for
 infrastructure dogfooding now (actually, yesterday), ready for feature
 tracking (and bugs tracking replacement for willing non-integrated
 guinea pigs) in 6 months, and ready for everyone in 12 months.
 
 -- 
 Thierry Carrez (ttx)
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] [neutron] local.conf for devstack using neutron on home network

[Kashyap Chamarthy]

On Sun, Nov 09, 2014 at 02:48:49PM +, Chris Dent wrote:
 On Sat, 8 Nov 2014, Kashyap Chamarthy wrote:
 
 [I realize you intend to use physical machine for DevStack, still I
 thought I'd post this here.]
 
 Thanks for posting it. Each added datapoint will get us closer.
 
 FWIW, this[1] is the minimal localrc contents (be sure to edit
 
 That's minimal? :)

Hmm, apart from Cinder service in the said config, rest of them all
noted there are essential to get a minimal, functional DevStack --
at-least in my testing. Cinder isn't normally part of my test
environment (maybe I should add it), but I was investigating a bug in
that case, so if you don't prefer it, you can elide these:
c-api,c-bak,c-sch,c-vol.

That said, I should have defined what I consider minimal, broadly: Nova
(libvirt/KVM driver with nested virt), Neutron (OVS+GRE or VXLAN),
Keystone (with PKI), Glance.

 Once the stack.sh is complete, I do some tasks Neutron expects:
 
  - Create a new private network
  - Create a new private subnet (on the above private network)
  - Create a router
  - Associate the router to an existing external network by setting it
as its gateway
  - Associate the private network interface to the router
  - Add Neutron security group rules for ICMP and SSH
 
 For devstack to live up to the dev in its name the above steps are
 something I would expect devstack to do for me, assuming I set the
 right varables and enabled the right services in local.conf.
 

-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] [neutron] local.conf for devstack using neutron on home network

[Kashyap Chamarthy]

On Thu, Nov 06, 2014 at 07:24:02PM +, Chris Dent wrote:
 
 I seem to be struggling to cook a neutron configuration for my little
 home network that doesn't end in a variety of failures (devstack errors
 out, floating ips are on an unexpected (and unrouted) network, etc).
 I recognize that neutron is going to be complex out of necessity, so
 I'm not complaining, I just don't know what to do.
 
 I've fought with this in a variety of ways, getting the sense that I'm
 just doing it completely wrong, so I thought perhaps I should just ask
 if someone can produce a config for my network (described below).
 
 If you do help I'll be eternally grateful and owe you virtual beer and
 you'll get the satisfaction of knowing you've helped to educate someone
 out of the pit of ignorance.
 
 I have a wifi network 192.168.1.0/24
 
 I have a mac mini on that network.
 
 I use internet sharing to have an ethernet network (192.168.2.0/24)
 behind the mac. Out the etherport is a switch with two additional hosts
 (2.2 and 2.3), on each of which I'd like to run devstack on bare-metal
 with floating ips on the 192.168.2 network.
 
 Internet sharing is satisfactory for me. I don't need to reach the
 devstack hosts or their guests from beyond the mac, but I would like to
 reach them from the mac.
 
 Each devstack host has one physical interface, eth0, with a static
 IP. I'd like compute instances to get floating ips from a portion of
 that network.
 
 Using nova-networking I can make this work without issue:
 
 ```
 [[local|localrc]]
 HOST_IP=192.168.2.3
 FLOATING_RANGE=192.168.2.128/26
 ```
 
 What transformation is needed to get similar functionality with
 neutron?

[I realize you intend to use physical machine for DevStack, still I
thought I'd post this here.]

FWIW, this[1] is the minimal localrc contents (be sure to edit
ENABLED_SERVICES config directive to fit your needs) I use in my
DevStack Neutron setup in a virtual machine. This setup uses nested KVM
(LIBVIRT_TYPE=kvm in localrc does it) -- for it work, the host needs to
have nested KVM enabled:

$ modinfo kvm_intel | grep -i nested
parm:   nested:boolkvm   435079  1 kvm_intel

More details on that here[2], in case anyone else finds it useful.

Once the stack.sh is complete, I do some tasks Neutron expects:

  - Create a new private network
  - Create a new private subnet (on the above private network)
  - Create a router
  - Associate the router to an existing external network by setting it
as its gateway
  - Associate the private network interface to the router
  - Add Neutron security group rules for ICMP and SSH


[1] 
https://kashyapc.fedorapeople.org/virt/openstack/minimal_devstack_localrc.txt
[2] 
http://kashyapc.fedorapeople.org/virt/procedure-to-enable-nested-virt-on-intel-machines.txt

-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cross distribution talks on Friday

[Kashyap Chamarthy]

On Sat, Nov 01, 2014 at 09:13:21PM +0800, Thomas Goirand wrote:
 Hi,
 
 I was wondering if some distribution OpenStack package maintainers
 would be interested to have some cross-distribution discussion on
 Friday, during the contributors sessions.
 
 I'll be happy to discuss with Ubuntu people, but not only. I'd like to
 meet also guys working with RedHat and Gentoo. I'm sure we may have
 some interesting things to share.

 
 OpenStack release management team would also be welcome.
 
 If you are interested, please reply to this mail.

You might also want to start an etherpad instance with some initial
agenda notes and throw the URL here to guage interest. 

On a related note, a bunch of Fedora/RHEL/CentOS/Scientific Linux
packagers are planning[1] to meetup at the summit to discuss RDO project
packaging aspects, etc.


  [1] https://etherpad.openstack.org/p/RDO_Meetup_Summit

-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Cross distribution talks on Friday

[Kashyap Chamarthy]

On Sat, Nov 01, 2014 at 01:45:24PM -0400, Adam Young wrote:
 On 11/01/2014 11:29 AM, Kashyap Chamarthy wrote:
 On Sat, Nov 01, 2014 at 09:13:21PM +0800, Thomas Goirand wrote:
 Hi,
 
 I was wondering if some distribution OpenStack package maintainers
 would be interested to have some cross-distribution discussion on
 Friday, during the contributors sessions.
 
 I'll be happy to discuss with Ubuntu people, but not only. I'd like to
 meet also guys working with RedHat and Gentoo. I'm sure we may have
 some interesting things to share.
 
 
 OpenStack release management team would also be welcome.
 
 If you are interested, please reply to this mail.
 You might also want to start an etherpad instance with some initial
 agenda notes and throw the URL here to guage interest.
 
 On a related note, a bunch of Fedora/RHEL/CentOS/Scientific Linux
 packagers are planning[1] to meetup at the summit to discuss RDO project
 packaging aspects, etc.
 
 
[1] https://etherpad.openstack.org/p/RDO_Meetup_Summit
 
 Getting the whole PBR version issues cleared up would be huge.

I took the liberty to add the above topic to the etherpad. If you'd like
to add more granular details, feel free to edit.

-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Summit] Coordination between OpenStack lower layer virt stack (libvirt, QEMU/KVM)

[Kashyap Chamarthy]

On Tue, Oct 21, 2014 at 12:07:41PM +0100, Daniel P. Berrange wrote:
 On Tue, Oct 21, 2014 at 12:58:48PM +0200, Kashyap Chamarthy wrote:
  I was discussing $subject on #openstack-nova, Nikola Dipanov suggested
  it's worthwhile to bring this up on the list.
  
  I was looking at 
  
  http://kilodesignsummit.sched.org/
  
  and noticed there's no specific session (correct me if I'm wrong) that's
  targeted at coordination between OpenStack - libvirt - QEMU/KVM.
 
 At previous summits, Nova has given each virt driver a dedicated session
 in its track. Those sessions have pretty much just been a walkthrough of
 the various features each virt team was planning.
 
 We always have far more topics to discuss than we have time available,
 and for this summit we want to change direction to maximise the value
 extracted from face-to-face meetings.
 
 As such any session which is just duplicating stuff that could easily be
 dealt with over email or irc is being cut, to make room for topics where
 we really need to have the f2f discussions. So the virt driver general
 sessions from previous summits are not likely to be on the schedule this
 time around.
 
Optimized for efficiency, good to know, thanks.

-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev