Re: [openstack-dev] [nova] bug triage experimentation

[Sean Dague]

On 07/20/2017 06:20 PM, Nematollah Bidokhti wrote:
> Hi,
> 
> I have missed the original email on this subject.
> We [Fault Genes WG] have been doing some machine learning analysis on Nova 
> bugs/issues from 3 different sources (Launchpad, Stackoverflow, 
> ask.openstack.org). We have been able to take all the issues and bring them 
> down to 15 clusters.
> We have tried to find open source tools that can help us define the fault 
> classifications, but have not been able to find any tool.
> 
> Therefore, our team have come to the conclusion that we need the support of 
> some Nova experts to help define the classifications. I would like to have 
> some discussions with Sean and others that have an interest in this area and 
> compare notes and see how we can collaborate.
> 
> The goal of our WG is to apply the same technique to all key OpenStack 
> projects.

Sure, would be happy to. All this went a little bit on hold as I was off
on vacation and a conference, and am now trying to help getting freeze
critical patches back in. But I'll probably start looking again more
deeply after the feature freeze.

If you can provide me pointers to your machine learning work that
clustered things, I'd happily take a look and see how that matches with
domain experts. Thanks a bunch for also diving in here!

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Nematollah Bidokhti]

Hi,

I have missed the original email on this subject.
We [Fault Genes WG] have been doing some machine learning analysis on Nova 
bugs/issues from 3 different sources (Launchpad, Stackoverflow, 
ask.openstack.org). We have been able to take all the issues and bring them 
down to 15 clusters.
We have tried to find open source tools that can help us define the fault 
classifications, but have not been able to find any tool.

Therefore, our team have come to the conclusion that we need the support of 
some Nova experts to help define the classifications. I would like to have some 
discussions with Sean and others that have an interest in this area and compare 
notes and see how we can collaborate.

The goal of our WG is to apply the same technique to all key OpenStack projects.

Thanks,
Nemat 

-Original Message-
From: Emilien Macchi [mailto:emil...@redhat.com] 
Sent: Wednesday, July 05, 2017 12:24 PM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [nova] bug triage experimentation

On Fri, Jun 23, 2017 at 9:52 AM, Sean Dague <s...@dague.net> wrote:
> The Nova bug backlog is just over 800 open bugs, which while 
> historically not terrible, remains too large to be collectively usable 
> to figure out where things stand. We've had a few recent issues where 
> we just happened to discover upgrade bugs filed 4 months ago that 
> needed fixes and backports.
>
> Historically we've tried to just solve the bug backlog with volunteers.
> We've had many a brave person dive into here, and burn out after 4 - 6 
> months. And we're currently without a bug lead. Having done a big 
> giant purge in the past
> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046
> 517.html)
> I know how daunting this all can be.
>
> I don't think that people can currently solve the bug triage problem 
> at the current workload that it creates. We've got to reduce the smart 
> human part of that workload.
>
> But, I think that we can also learn some lessons from what active 
> github projects do.
>
> #1 Bot away bad states
>
> There are known bad states of bugs - In Progress with no open patch, 
> Assigned but not In Progress. We can just bot these away with scripts.
> Even better would be to react immediately on bugs like those, that 
> helps to train folks how to use our workflow. I've got some starter 
> scripts for this up at - https://github.com/sdague/nova-bug-tools
>
> #2 Use tag based workflow
>
> One lesson from github projects, is the github tracker has no workflow.
> Issues are openned or closed. Workflow has to be invented by every 
> team based on a set of tags. Sometimes that's annoying, but often 
> times it's super handy, because it allows the tracker to change 
> workflows and not try to change the meaning of things like "Confirmed 
> vs. Triaged" in your mind.
>
> We can probably tag for information we know we need at lot easier. I'm 
> considering something like
>
> * needs.system-version
> * needs.openstack-version
> * needs.logs
> * needs.subteam-feedback
> * has.system-version
> * has.openstack-version
> * has.reproduce
>
> Some of these a bot can process the text on and tell if that info was 
> provided, and comment how to provide the updated info. Some of this 
> would be human, but with official tags, it would probably help.
>
> #3 machine assisted functional tagging
>
> I'm playing around with some things that might be useful in mapping 
> new bugs into existing functional buckets like: libvirt, volumes, etc. 
> We'll see how useful it ends up being.
>
> #4 reporting on smaller slices
>
> Build some tooling to report on the status and change over time of 
> bugs under various tags. This will help visualize how we are doing
> (hopefully) and where the biggest piles of issues are.
>
> The intent is the normal unit of interaction would be one of these 
> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36 
> vmware bugs. It would also highlight the rates of change in these 
> piles, and what's getting attention and what is not.
>
>
> This is going to be kind of an ongoing experiment, but as we currently 
> have no one spear heading bug triage, it seemed like a good time to 
> try this out.
>
> Comments and other suggestions are welcomed. The tooling will have the 
> nova flow in mind, but I'm trying to make it so it takes a project 
> name as params on all the scripts, so anyone can use it. It's a little 
> hack and slash right now to discover what the right patterns are.

I also believe that some of the scripts could be transformed into native 
features of Storyboard where bugs could be auto-triaged periodically without 
human

Re: [openstack-dev] [nova] bug triage experimentation

[Jeremy Stanley]

On 2017-07-12 15:50:26 -0500 (-0500), Jay S Bryant wrote:
[...]
> Looking for the, 'here is what everyone needs to know to start
> using StoryBoard' documentation so I can get that linked in our
> devref.
[...]

An excellent point, and as always a bit of a challenge when
designing software for use within and outside our community to
separate documentation on "how to use X" from "how the OpenStack
community uses X." There have been a number of summit/forum
presentations and some excellent blog posts (particularly at
https://storyboard-blog.io/ ) which touch on these subjects, but of
course those are point-in-time anecdotal prose unsuited to treatment
as long-term reference documentation.

The general idea, I think, is that documentation on using the WebUI
should be readily accessible hosted alongside storyboard-webclient,
e.g. https://storyboard.openstack.org/#!/page/about (which is
realistically just a stub for now). I think it's up for debate
whether its API documentation should be there, or where it lives now
at https://docs.openstack.org/infra/storyboard/ or maybe both. And
then workflow documentation specific to our community probably
belongs somewhere like https://docs.openstack.org/infra/manual/
(there's a little bit in the Developer’s Guide there under
Development Workflow, but as always it could use expanding on).

So in summary, like with any software the documentation is
perpetually insufficient but hopefully something we can all expand
on together given adequate experience and feedback.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Jeremy Stanley]

On 2017-07-12 14:36:21 -0600 (-0600), Sean Dague wrote:
[...]
> It would be really nice if we could this as first class support
> and not bolt on later, because I fear we're going to get pretty
> complicated adhoc schema definitions here that have to be done
> client side.

It's still not entirely clear to me what you're looking for. Like in
LP, tags aren't applied to individual tasks. In LP they're applied
to entire bugs or in SB they're applied to entire stories. Like LP
bugs, stories in SB can have tasks from multiple "projects" (which
in our case with SB map to individual Git repos). Are you requesting
some sort of namespace/prefix selector surfaced in the WebUI when
setting tag names on a story? And if so, where do we take those
namespaces from?

It seems like as a community we can just have a convention that if
someone sets a nova-foo tag on a random story that has nothing to do
with the semantics the nova team expects around the nova-foo tag,
then when that shows up in the Nova team's queries they can unset
that tag on the offending story without worry they're stepping on
anyone else's toes.

The only alternative I can think of is that we, as OpenStack, decide
how we will collectively perform all story tagging. Individual teams
have preferred to use their own workflows for defect tracking in the
past, so if they're all going to play together in a cross-project
system they either need to agree on what different tags mean or
agree to all use distinct tags. I can only assume we'll run into the
same problem in Gerrit with global review tags too, once we have
them.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Jay S Bryant]

On 7/12/2017 3:44 PM, Kendall Nelson wrote:

@Jay: I would definitely urge projects to update the places they have 
any info about bug tracking since different projects use tags 
differently. I suppose we could add to the SB documentation what the 
common practices are, but I imagine there will be a lot of project 
specific details about how the project sets up their boards and 
worklists, what tags they use, how things are formatted etc that would 
be better kept elsewhere.


Agreed, this is an opportunity to for Cinder to document how we use 
StoryBoard as we migrate to it and anything that is specific to our 
usage is something we document.  :-)  I am asking if there is a high 
level common documentation location.  I found development documentation 
and links in your Wiki.  Looking for the, 'here is what everyone needs 
to know to start using StoryBoard' documentation so I can get that 
linked in our devref.


Hope that makes sense.

@Sean: Agreed! Having it dealt with before we reach critical mass in 
SB will save us later. As one of the people helping people migrate to 
sb, I will include this decision in the information I give projects 
that are migrating. I think we have a FAQ somewhere we can add the 
decision to as well.


-Kendall (diablo_rojo)

On Wed, Jul 12, 2017 at 1:34 PM Jay S Bryant > wrote:


Kendall,

It looks like our current bug tracking documentation is quite
minimal:
https://docs.openstack.org/cinder/latest/devref/launchpad.html#bug-tracking

Is there going to be a place where SB is going to be documented
with some of these details that we can link to under our
bug-tracking section?

Thanks!

Jay



On 7/12/2017 3:19 PM, Kendall Nelson wrote:

Hey Sean :)

So we discussed the issue of tag collisions in the SB meeting we
had today. Basically, we came to the conclusion that projects
should append their project to the start of the tag, thereby
avoiding collision i.e. ironic-compute, nova-compute,
manila-storage, swift-storage, cinder-storage. If we can ask bug
triagers in their respective projects to follow and uphold the
convention, we should be fine. It might also be helpful to add
this to any directions projects might have about filing bugs so
new contributors start off on the right foot.

Thanks for bringing this concern up before it becomes a problem!
If anyone has other questions or concerns, please attend our
meetings or drop into our channel (#storyboard)!

-Kendall Nelson(diablo_rojo)

[1] https://wiki.openstack.org/wiki/Meetings/StoryBoard

On Wed, Jul 12, 2017 at 5:47 AM Sean Dague > wrote:

On 07/11/2017 04:31 PM, Jeremy Stanley wrote:
> On 2017-07-10 07:33:28 -0400 (-0400), Sean Dague wrote:
> [...]
>> Ideally storyboard would just be a lot more receptive to
these kinds of
>> things, by emitting a more native event stream,
>
> Well, there is
> http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/notifications/publisher.py
>
> so replacing or partnering its RabbitMQ publisher with
something
> like an MQTT publisher into firehose.openstack.org
 is probably not
> terribly hard for someone with interest in that and would be
> generally useful.
>
>> and having really good tag support (preferably actually
project
>> scoped tags, so setting it on the nova task doesn't impact the
>> neutron tasks on the same story, as an for instance)
> [...]
>
> Your queries (including those used to build automatic
tasklists and
> boards) could just include project in addition to tag,
right? Or is
> this more of a UI concern, being able to click on an
arbitrary tag
> in the webclient and only get back a set of tagged stories
for the
> same project rather than across all projects?

My concern is based on current limitations in launchpad, and
to make
sure they don't get encoded into Storyboard.

Tags in launchpad are at the Bug level. Bugs map to projects
as Tasks.
Which is why you can have 1 Bug set to be impacting both Nova and
Neutron. You get lots of weirdness today when for instance a
bug is
assigned to Nova and Ironic, and the Nova team tags it
"ironic" in
triage, but that means that now Ironic has a bug with the
"ironic" tag.
Then if later Nova is removed from the bug, it ends up really all
looking odd and confusing.

Or the fact that "compute" as a Nova tag means the compute
worker, but
other teams tag things with compute to just mean Nova is
involved.

Re: [openstack-dev] [nova] bug triage experimentation

[Kendall Nelson]

@Jay: I would definitely urge projects to update the places they have any
info about bug tracking since different projects use tags differently. I
suppose we could add to the SB documentation what the common practices are,
but I imagine there will be a lot of project specific details about how the
project sets up their boards and worklists, what tags they use, how things
are formatted etc that would be better kept elsewhere.

@Sean: Agreed! Having it dealt with before we reach critical mass in SB
will save us later. As one of the people helping people migrate to sb, I
will include this decision in the information I give projects that are
migrating. I think we have a FAQ somewhere we can add the decision to as
well.

-Kendall (diablo_rojo)

On Wed, Jul 12, 2017 at 1:34 PM Jay S Bryant  wrote:

> Kendall,
>
> It looks like our current bug tracking documentation is quite minimal:
> https://docs.openstack.org/cinder/latest/devref/launchpad.html#bug-tracking
>
> Is there going to be a place where SB is going to be documented with some
> of these details that we can link to under our bug-tracking section?
>
> Thanks!
>
> Jay
>
>
>
> On 7/12/2017 3:19 PM, Kendall Nelson wrote:
>
> Hey Sean :)
>
> So we discussed the issue of tag collisions in the SB meeting we had
> today. Basically, we came to the conclusion that projects should append
> their project to the start of the tag, thereby avoiding collision i.e.
> ironic-compute, nova-compute, manila-storage, swift-storage,
> cinder-storage. If we can ask bug triagers in their respective projects to
> follow and uphold the convention, we should be fine. It might also be
> helpful to add this to any directions projects might have about filing bugs
> so new contributors start off on the right foot.
>
> Thanks for bringing this concern up before it becomes a problem! If anyone
> has other questions or concerns, please attend our meetings or drop into
> our channel (#storyboard)!
>
> -Kendall Nelson(diablo_rojo)
>
> [1] https://wiki.openstack.org/wiki/Meetings/StoryBoard
>
> On Wed, Jul 12, 2017 at 5:47 AM Sean Dague  wrote:
>
>> On 07/11/2017 04:31 PM, Jeremy Stanley wrote:
>> > On 2017-07-10 07:33:28 -0400 (-0400), Sean Dague wrote:
>> > [...]
>> >> Ideally storyboard would just be a lot more receptive to these kinds of
>> >> things, by emitting a more native event stream,
>> >
>> > Well, there is
>> > > http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/notifications/publisher.py
>> >
>> > so replacing or partnering its RabbitMQ publisher with something
>> > like an MQTT publisher into firehose.openstack.org is probably not
>> > terribly hard for someone with interest in that and would be
>> > generally useful.
>> >
>> >> and having really good tag support (preferably actually project
>> >> scoped tags, so setting it on the nova task doesn't impact the
>> >> neutron tasks on the same story, as an for instance)
>> > [...]
>> >
>> > Your queries (including those used to build automatic tasklists and
>> > boards) could just include project in addition to tag, right? Or is
>> > this more of a UI concern, being able to click on an arbitrary tag
>> > in the webclient and only get back a set of tagged stories for the
>> > same project rather than across all projects?
>>
>> My concern is based on current limitations in launchpad, and to make
>> sure they don't get encoded into Storyboard.
>>
>> Tags in launchpad are at the Bug level. Bugs map to projects as Tasks.
>> Which is why you can have 1 Bug set to be impacting both Nova and
>> Neutron. You get lots of weirdness today when for instance a bug is
>> assigned to Nova and Ironic, and the Nova team tags it "ironic" in
>> triage, but that means that now Ironic has a bug with the "ironic" tag.
>> Then if later Nova is removed from the bug, it ends up really all
>> looking odd and confusing.
>>
>> Or the fact that "compute" as a Nova tag means the compute worker, but
>> other teams tag things with compute to just mean Nova is involved.
>> Project scoped tags would help clarify what context it is in.
>>
>> -Sean
>>
>> >
>> >
>> >
>> >
>> __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> 

Re: [openstack-dev] [nova] bug triage experimentation

[Sean Dague]

On 07/12/2017 02:19 PM, Kendall Nelson wrote:

Hey Sean :)

So we discussed the issue of tag collisions in the SB meeting we had 
today. Basically, we came to the conclusion that projects should append 
their project to the start of the tag, thereby avoiding collision i.e. 
ironic-compute, nova-compute, manila-storage, swift-storage, 
cinder-storage. If we can ask bug triagers in their respective projects 
to follow and uphold the convention, we should be fine. It might also be 
helpful to add this to any directions projects might have about filing 
bugs so new contributors start off on the right foot.


It would be really nice if we could this as first class support and not 
bolt on later, because I fear we're going to get pretty complicated 
adhoc schema definitions here that have to be done client side.


-Sean

--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Jay S Bryant]

Kendall,

It looks like our current bug tracking documentation is quite minimal: 
https://docs.openstack.org/cinder/latest/devref/launchpad.html#bug-tracking


Is there going to be a place where SB is going to be documented with 
some of these details that we can link to under our bug-tracking section?


Thanks!

Jay



On 7/12/2017 3:19 PM, Kendall Nelson wrote:

Hey Sean :)

So we discussed the issue of tag collisions in the SB meeting we had 
today. Basically, we came to the conclusion that projects should 
append their project to the start of the tag, thereby avoiding 
collision i.e. ironic-compute, nova-compute, manila-storage, 
swift-storage, cinder-storage. If we can ask bug triagers in their 
respective projects to follow and uphold the convention, we should be 
fine. It might also be helpful to add this to any directions projects 
might have about filing bugs so new contributors start off on the 
right foot.


Thanks for bringing this concern up before it becomes a problem! If 
anyone has other questions or concerns, please attend our meetings or 
drop into our channel (#storyboard)!


-Kendall Nelson(diablo_rojo)

[1] https://wiki.openstack.org/wiki/Meetings/StoryBoard

On Wed, Jul 12, 2017 at 5:47 AM Sean Dague > wrote:


On 07/11/2017 04:31 PM, Jeremy Stanley wrote:
> On 2017-07-10 07:33:28 -0400 (-0400), Sean Dague wrote:
> [...]
>> Ideally storyboard would just be a lot more receptive to these
kinds of
>> things, by emitting a more native event stream,
>
> Well, there is
> http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/notifications/publisher.py
>
> so replacing or partnering its RabbitMQ publisher with something
> like an MQTT publisher into firehose.openstack.org
 is probably not
> terribly hard for someone with interest in that and would be
> generally useful.
>
>> and having really good tag support (preferably actually project
>> scoped tags, so setting it on the nova task doesn't impact the
>> neutron tasks on the same story, as an for instance)
> [...]
>
> Your queries (including those used to build automatic tasklists and
> boards) could just include project in addition to tag, right? Or is
> this more of a UI concern, being able to click on an arbitrary tag
> in the webclient and only get back a set of tagged stories for the
> same project rather than across all projects?

My concern is based on current limitations in launchpad, and to make
sure they don't get encoded into Storyboard.

Tags in launchpad are at the Bug level. Bugs map to projects as Tasks.
Which is why you can have 1 Bug set to be impacting both Nova and
Neutron. You get lots of weirdness today when for instance a bug is
assigned to Nova and Ironic, and the Nova team tags it "ironic" in
triage, but that means that now Ironic has a bug with the "ironic"
tag.
Then if later Nova is removed from the bug, it ends up really all
looking odd and confusing.

Or the fact that "compute" as a Nova tag means the compute worker, but
other teams tag things with compute to just mean Nova is involved.
Project scoped tags would help clarify what context it is in.

-Sean

>
>
>
>
__
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Kendall Nelson]

Hey Sean :)

So we discussed the issue of tag collisions in the SB meeting we had today.
Basically, we came to the conclusion that projects should append their
project to the start of the tag, thereby avoiding collision i.e.
ironic-compute, nova-compute, manila-storage, swift-storage,
cinder-storage. If we can ask bug triagers in their respective projects to
follow and uphold the convention, we should be fine. It might also be
helpful to add this to any directions projects might have about filing bugs
so new contributors start off on the right foot.

Thanks for bringing this concern up before it becomes a problem! If anyone
has other questions or concerns, please attend our meetings or drop into
our channel (#storyboard)!

-Kendall Nelson(diablo_rojo)

[1] https://wiki.openstack.org/wiki/Meetings/StoryBoard

On Wed, Jul 12, 2017 at 5:47 AM Sean Dague  wrote:

> On 07/11/2017 04:31 PM, Jeremy Stanley wrote:
> > On 2017-07-10 07:33:28 -0400 (-0400), Sean Dague wrote:
> > [...]
> >> Ideally storyboard would just be a lot more receptive to these kinds of
> >> things, by emitting a more native event stream,
> >
> > Well, there is
> >  http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/notifications/publisher.py
> >
> > so replacing or partnering its RabbitMQ publisher with something
> > like an MQTT publisher into firehose.openstack.org is probably not
> > terribly hard for someone with interest in that and would be
> > generally useful.
> >
> >> and having really good tag support (preferably actually project
> >> scoped tags, so setting it on the nova task doesn't impact the
> >> neutron tasks on the same story, as an for instance)
> > [...]
> >
> > Your queries (including those used to build automatic tasklists and
> > boards) could just include project in addition to tag, right? Or is
> > this more of a UI concern, being able to click on an arbitrary tag
> > in the webclient and only get back a set of tagged stories for the
> > same project rather than across all projects?
>
> My concern is based on current limitations in launchpad, and to make
> sure they don't get encoded into Storyboard.
>
> Tags in launchpad are at the Bug level. Bugs map to projects as Tasks.
> Which is why you can have 1 Bug set to be impacting both Nova and
> Neutron. You get lots of weirdness today when for instance a bug is
> assigned to Nova and Ironic, and the Nova team tags it "ironic" in
> triage, but that means that now Ironic has a bug with the "ironic" tag.
> Then if later Nova is removed from the bug, it ends up really all
> looking odd and confusing.
>
> Or the fact that "compute" as a Nova tag means the compute worker, but
> other teams tag things with compute to just mean Nova is involved.
> Project scoped tags would help clarify what context it is in.
>
> -Sean
>
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Sean Dague]

On 07/11/2017 04:31 PM, Jeremy Stanley wrote:

On 2017-07-10 07:33:28 -0400 (-0400), Sean Dague wrote:
[...]

Ideally storyboard would just be a lot more receptive to these kinds of
things, by emitting a more native event stream,


Well, there is
http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/notifications/publisher.py
 >
so replacing or partnering its RabbitMQ publisher with something
like an MQTT publisher into firehose.openstack.org is probably not
terribly hard for someone with interest in that and would be
generally useful.


and having really good tag support (preferably actually project
scoped tags, so setting it on the nova task doesn't impact the
neutron tasks on the same story, as an for instance)

[...]

Your queries (including those used to build automatic tasklists and
boards) could just include project in addition to tag, right? Or is
this more of a UI concern, being able to click on an arbitrary tag
in the webclient and only get back a set of tagged stories for the
same project rather than across all projects?


My concern is based on current limitations in launchpad, and to make 
sure they don't get encoded into Storyboard.


Tags in launchpad are at the Bug level. Bugs map to projects as Tasks. 
Which is why you can have 1 Bug set to be impacting both Nova and 
Neutron. You get lots of weirdness today when for instance a bug is 
assigned to Nova and Ironic, and the Nova team tags it "ironic" in 
triage, but that means that now Ironic has a bug with the "ironic" tag. 
Then if later Nova is removed from the bug, it ends up really all 
looking odd and confusing.


Or the fact that "compute" as a Nova tag means the compute worker, but 
other teams tag things with compute to just mean Nova is involved. 
Project scoped tags would help clarify what context it is in.


-Sean





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Jeremy Stanley]

On 2017-07-10 07:33:28 -0400 (-0400), Sean Dague wrote:
[...]
> Ideally storyboard would just be a lot more receptive to these kinds of
> things, by emitting a more native event stream,

Well, there is
http://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/notifications/publisher.py
 >
so replacing or partnering its RabbitMQ publisher with something
like an MQTT publisher into firehose.openstack.org is probably not
terribly hard for someone with interest in that and would be
generally useful.

> and having really good tag support (preferably actually project
> scoped tags, so setting it on the nova task doesn't impact the
> neutron tasks on the same story, as an for instance)
[...]

Your queries (including those used to build automatic tasklists and
boards) could just include project in addition to tag, right? Or is
this more of a UI concern, being able to click on an arbitrary tag
in the webclient and only get back a set of tagged stories for the
same project rather than across all projects?
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[James E. Blair]

Sean Dague  writes:

> On 07/05/2017 03:23 PM, Emilien Macchi wrote:
> 
>> 
>> I also believe that some of the scripts could be transformed into
>> native features of Storyboard where bugs could be auto-triaged
>> periodically without human intervention.
>> Maybe it would convince more OpenStack projects to leave Launchpad and
>> adopt Storyboard?
>> I would certainly one of those and propose such a change for TripleO &
>> related projects.
>
> Maybe... my concern there is that workflow encoded into trackers is
> pretty static, and it's hard to evolve, because it impacts all users of
> that platform. Where as a script that processes bugs externally can
> adapt really quickly based on what's working / not working with a
> particular team. There is no 1 right way to handle bugs, it's just about
> making every bug handling team the most effective that they can be.
> Which means I assume that different teams would find different parts of
> this useful, and other parts things they wouldn't want to use at all.
> That's why I tried to make every "processing unit" it's own cli.
>
> Ideally storyboard would just be a lot more receptive to these kinds of
> things, by emitting a more native event stream, and having really good
> tag support (preferably actually project scoped tags, so setting it on
> the nova task doesn't impact the neutron tasks on the same story, as an
> for instance) so the hack we need to do on LP isn't needed. But,
> actually, beyond that, keeping the processing logic team specific is a
> good thing. It's much like the fact that we've largely done gerrit
> review dashboards client side, because they are fast to iterate on, then
> server side.

I agree.  I think being able to add things to Storyboard is great, and
as we've been using it more, we've done some of that.  But we've also
run into places where we found that we needed Storyboard to do some
things that were ultimately project-specific workflows.  So I think long
term we're going to have both things -- adding features that make sense
globally as well as ones that facilitate local configuration and
workflows.

As an example, the "board" feature on storyboard can be really useful,
but we wanted to automate some of the movement between lanes.  Lanes are
arbitrary.  Rather than writing a new processing language to describe
that and incorporating that into Storyboard, we wrote a script to manage
one specific board using the Storyboard API.

The board is here: https://storyboard.openstack.org/#!/board/41

The script is here: 
http://git.openstack.org/cgit/openstack-infra/zuul/tree/tools/update-storyboard.py?h=feature/zuulv3

(Basically, that script automatically moves tasks between lanes based on
status according to the map defined on line 65, while still allowing
folks to manually move tasks between certain classes of lanes -- so a
task marked as 'todo' can be in either the 'New', 'Backlog', or 'Todo'
lanes.)

I'm imagining a future where we have lots of scripts like that (or maybe
a few framework scripts like Sean's, with configuration), and we run
those scripts in Infra but projects are responsible for their own
configuration.

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Sean Dague]

On 07/05/2017 03:23 PM, Emilien Macchi wrote:

> 
> I also believe that some of the scripts could be transformed into
> native features of Storyboard where bugs could be auto-triaged
> periodically without human intervention.
> Maybe it would convince more OpenStack projects to leave Launchpad and
> adopt Storyboard?
> I would certainly one of those and propose such a change for TripleO &
> related projects.

Maybe... my concern there is that workflow encoded into trackers is
pretty static, and it's hard to evolve, because it impacts all users of
that platform. Where as a script that processes bugs externally can
adapt really quickly based on what's working / not working with a
particular team. There is no 1 right way to handle bugs, it's just about
making every bug handling team the most effective that they can be.
Which means I assume that different teams would find different parts of
this useful, and other parts things they wouldn't want to use at all.
That's why I tried to make every "processing unit" it's own cli.

Ideally storyboard would just be a lot more receptive to these kinds of
things, by emitting a more native event stream, and having really good
tag support (preferably actually project scoped tags, so setting it on
the nova task doesn't impact the neutron tasks on the same story, as an
for instance) so the hack we need to do on LP isn't needed. But,
actually, beyond that, keeping the processing logic team specific is a
good thing. It's much like the fact that we've largely done gerrit
review dashboards client side, because they are fast to iterate on, then
server side.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Sean Dague]

On 07/05/2017 03:07 PM, Emilien Macchi wrote:
> On Wed, Jun 28, 2017 at 7:33 AM, Ben Nemec  wrote:
>>
>>
>> On 06/23/2017 11:52 AM, Sean Dague wrote:
>>>
>>> The Nova bug backlog is just over 800 open bugs, which while
>>> historically not terrible, remains too large to be collectively usable
>>> to figure out where things stand. We've had a few recent issues where we
>>> just happened to discover upgrade bugs filed 4 months ago that needed
>>> fixes and backports.
>>>
>>> Historically we've tried to just solve the bug backlog with volunteers.
>>> We've had many a brave person dive into here, and burn out after 4 - 6
>>> months. And we're currently without a bug lead. Having done a big giant
>>> purge in the past
>>>
>>> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
>>> I know how daunting this all can be.
>>>
>>> I don't think that people can currently solve the bug triage problem at
>>> the current workload that it creates. We've got to reduce the smart
>>> human part of that workload.
>>>
>>> But, I think that we can also learn some lessons from what active github
>>> projects do.
>>>
>>> #1 Bot away bad states
>>>
>>> There are known bad states of bugs - In Progress with no open patch,
>>> Assigned but not In Progress. We can just bot these away with scripts.
>>> Even better would be to react immediately on bugs like those, that helps
>>> to train folks how to use our workflow. I've got some starter scripts
>>> for this up at - https://github.com/sdague/nova-bug-tools
>>
>>
>> Just saw the update on https://bugs.launchpad.net/nova/+bug/1698010 and I
>> don't agree that assigned but not in progress is an invalid state.  If it
>> persists for a period of time then sure, but to me assigning yourself a bug
>> is a signal that you're working on it and nobody else needs to. Otherwise
>> you end up with multiple people working a bug without realizing someone else
>> already was.  I've seen that happen more than once.
>>
>> Would it be possible to only un-assign such bugs if they've been in that
>> state for a week?  At that point it seems safe to say the assignee has
>> either moved on or that the bug is tricky and additional input would be
>> useful anyway.
>>
>> Otherwise, big +1 to a bug bot.  I need to try running it against the ~700
>> open TripleO bugs...
> 
> I just tried, please send complains to me if I broke something.
> 
> Sean, the tool is really awesome and I was wondering if we could move
> it to https://github.com/openstack-infra/release-tools so we
> centralize the tools.

Probably yes, eventually? Right now I'm honestly trying to figure out
the things that are useful here, and the ones that aren't, so a more
fluid experimentation is worth while.

My end game is to get some of these running and responding in real time
which means the core processing logic is going to evolve away from
scripts and into some kind of function plugin (the batch interface will
just still call them).

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Emilien Macchi]

On Fri, Jun 23, 2017 at 9:52 AM, Sean Dague  wrote:
> The Nova bug backlog is just over 800 open bugs, which while
> historically not terrible, remains too large to be collectively usable
> to figure out where things stand. We've had a few recent issues where we
> just happened to discover upgrade bugs filed 4 months ago that needed
> fixes and backports.
>
> Historically we've tried to just solve the bug backlog with volunteers.
> We've had many a brave person dive into here, and burn out after 4 - 6
> months. And we're currently without a bug lead. Having done a big giant
> purge in the past
> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
> I know how daunting this all can be.
>
> I don't think that people can currently solve the bug triage problem at
> the current workload that it creates. We've got to reduce the smart
> human part of that workload.
>
> But, I think that we can also learn some lessons from what active github
> projects do.
>
> #1 Bot away bad states
>
> There are known bad states of bugs - In Progress with no open patch,
> Assigned but not In Progress. We can just bot these away with scripts.
> Even better would be to react immediately on bugs like those, that helps
> to train folks how to use our workflow. I've got some starter scripts
> for this up at - https://github.com/sdague/nova-bug-tools
>
> #2 Use tag based workflow
>
> One lesson from github projects, is the github tracker has no workflow.
> Issues are openned or closed. Workflow has to be invented by every team
> based on a set of tags. Sometimes that's annoying, but often times it's
> super handy, because it allows the tracker to change workflows and not
> try to change the meaning of things like "Confirmed vs. Triaged" in your
> mind.
>
> We can probably tag for information we know we need at lot easier. I'm
> considering something like
>
> * needs.system-version
> * needs.openstack-version
> * needs.logs
> * needs.subteam-feedback
> * has.system-version
> * has.openstack-version
> * has.reproduce
>
> Some of these a bot can process the text on and tell if that info was
> provided, and comment how to provide the updated info. Some of this
> would be human, but with official tags, it would probably help.
>
> #3 machine assisted functional tagging
>
> I'm playing around with some things that might be useful in mapping new
> bugs into existing functional buckets like: libvirt, volumes, etc. We'll
> see how useful it ends up being.
>
> #4 reporting on smaller slices
>
> Build some tooling to report on the status and change over time of bugs
> under various tags. This will help visualize how we are doing
> (hopefully) and where the biggest piles of issues are.
>
> The intent is the normal unit of interaction would be one of these
> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36
> vmware bugs. It would also highlight the rates of change in these piles,
> and what's getting attention and what is not.
>
>
> This is going to be kind of an ongoing experiment, but as we currently
> have no one spear heading bug triage, it seemed like a good time to try
> this out.
>
> Comments and other suggestions are welcomed. The tooling will have the
> nova flow in mind, but I'm trying to make it so it takes a project name
> as params on all the scripts, so anyone can use it. It's a little hack
> and slash right now to discover what the right patterns are.

I also believe that some of the scripts could be transformed into
native features of Storyboard where bugs could be auto-triaged
periodically without human intervention.
Maybe it would convince more OpenStack projects to leave Launchpad and
adopt Storyboard?
I would certainly one of those and propose such a change for TripleO &
related projects.

Thanks,

> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Emilien Macchi]

On Wed, Jun 28, 2017 at 7:33 AM, Ben Nemec  wrote:
>
>
> On 06/23/2017 11:52 AM, Sean Dague wrote:
>>
>> The Nova bug backlog is just over 800 open bugs, which while
>> historically not terrible, remains too large to be collectively usable
>> to figure out where things stand. We've had a few recent issues where we
>> just happened to discover upgrade bugs filed 4 months ago that needed
>> fixes and backports.
>>
>> Historically we've tried to just solve the bug backlog with volunteers.
>> We've had many a brave person dive into here, and burn out after 4 - 6
>> months. And we're currently without a bug lead. Having done a big giant
>> purge in the past
>>
>> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
>> I know how daunting this all can be.
>>
>> I don't think that people can currently solve the bug triage problem at
>> the current workload that it creates. We've got to reduce the smart
>> human part of that workload.
>>
>> But, I think that we can also learn some lessons from what active github
>> projects do.
>>
>> #1 Bot away bad states
>>
>> There are known bad states of bugs - In Progress with no open patch,
>> Assigned but not In Progress. We can just bot these away with scripts.
>> Even better would be to react immediately on bugs like those, that helps
>> to train folks how to use our workflow. I've got some starter scripts
>> for this up at - https://github.com/sdague/nova-bug-tools
>
>
> Just saw the update on https://bugs.launchpad.net/nova/+bug/1698010 and I
> don't agree that assigned but not in progress is an invalid state.  If it
> persists for a period of time then sure, but to me assigning yourself a bug
> is a signal that you're working on it and nobody else needs to. Otherwise
> you end up with multiple people working a bug without realizing someone else
> already was.  I've seen that happen more than once.

I agree with you Ben. While I was running this query for old bug, I've
stopped so bugs after March of this year won't be modified (let me
know if that's the case, then I'll fix it).

A grace period of 7 days is a good idea maybe.

> Would it be possible to only un-assign such bugs if they've been in that
> state for a week?  At that point it seems safe to say the assignee has
> either moved on or that the bug is tricky and additional input would be
> useful anyway.
>
> Otherwise, big +1 to a bug bot.  I need to try running it against the ~700
> open TripleO bugs...
>
>
>>
>> #2 Use tag based workflow
>>
>> One lesson from github projects, is the github tracker has no workflow.
>> Issues are openned or closed. Workflow has to be invented by every team
>> based on a set of tags. Sometimes that's annoying, but often times it's
>> super handy, because it allows the tracker to change workflows and not
>> try to change the meaning of things like "Confirmed vs. Triaged" in your
>> mind.
>>
>> We can probably tag for information we know we need at lot easier. I'm
>> considering something like
>>
>> * needs.system-version
>> * needs.openstack-version
>> * needs.logs
>> * needs.subteam-feedback
>> * has.system-version
>> * has.openstack-version
>> * has.reproduce
>>
>> Some of these a bot can process the text on and tell if that info was
>> provided, and comment how to provide the updated info. Some of this
>> would be human, but with official tags, it would probably help.
>>
>> #3 machine assisted functional tagging
>>
>> I'm playing around with some things that might be useful in mapping new
>> bugs into existing functional buckets like: libvirt, volumes, etc. We'll
>> see how useful it ends up being.
>>
>> #4 reporting on smaller slices
>>
>> Build some tooling to report on the status and change over time of bugs
>> under various tags. This will help visualize how we are doing
>> (hopefully) and where the biggest piles of issues are.
>>
>> The intent is the normal unit of interaction would be one of these
>> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36
>> vmware bugs. It would also highlight the rates of change in these piles,
>> and what's getting attention and what is not.
>>
>>
>> This is going to be kind of an ongoing experiment, but as we currently
>> have no one spear heading bug triage, it seemed like a good time to try
>> this out.
>>
>> Comments and other suggestions are welcomed. The tooling will have the
>> nova flow in mind, but I'm trying to make it so it takes a project name
>> as params on all the scripts, so anyone can use it. It's a little hack
>> and slash right now to discover what the right patterns are.
>>
>> -Sean
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

Re: [openstack-dev] [nova] bug triage experimentation

[Emilien Macchi]

On Wed, Jun 28, 2017 at 7:33 AM, Ben Nemec  wrote:
>
>
> On 06/23/2017 11:52 AM, Sean Dague wrote:
>>
>> The Nova bug backlog is just over 800 open bugs, which while
>> historically not terrible, remains too large to be collectively usable
>> to figure out where things stand. We've had a few recent issues where we
>> just happened to discover upgrade bugs filed 4 months ago that needed
>> fixes and backports.
>>
>> Historically we've tried to just solve the bug backlog with volunteers.
>> We've had many a brave person dive into here, and burn out after 4 - 6
>> months. And we're currently without a bug lead. Having done a big giant
>> purge in the past
>>
>> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
>> I know how daunting this all can be.
>>
>> I don't think that people can currently solve the bug triage problem at
>> the current workload that it creates. We've got to reduce the smart
>> human part of that workload.
>>
>> But, I think that we can also learn some lessons from what active github
>> projects do.
>>
>> #1 Bot away bad states
>>
>> There are known bad states of bugs - In Progress with no open patch,
>> Assigned but not In Progress. We can just bot these away with scripts.
>> Even better would be to react immediately on bugs like those, that helps
>> to train folks how to use our workflow. I've got some starter scripts
>> for this up at - https://github.com/sdague/nova-bug-tools
>
>
> Just saw the update on https://bugs.launchpad.net/nova/+bug/1698010 and I
> don't agree that assigned but not in progress is an invalid state.  If it
> persists for a period of time then sure, but to me assigning yourself a bug
> is a signal that you're working on it and nobody else needs to. Otherwise
> you end up with multiple people working a bug without realizing someone else
> already was.  I've seen that happen more than once.
>
> Would it be possible to only un-assign such bugs if they've been in that
> state for a week?  At that point it seems safe to say the assignee has
> either moved on or that the bug is tricky and additional input would be
> useful anyway.
>
> Otherwise, big +1 to a bug bot.  I need to try running it against the ~700
> open TripleO bugs...

I just tried, please send complains to me if I broke something.

Sean, the tool is really awesome and I was wondering if we could move
it to https://github.com/openstack-infra/release-tools so we
centralize the tools.

Thanks,

>
>>
>> #2 Use tag based workflow
>>
>> One lesson from github projects, is the github tracker has no workflow.
>> Issues are openned or closed. Workflow has to be invented by every team
>> based on a set of tags. Sometimes that's annoying, but often times it's
>> super handy, because it allows the tracker to change workflows and not
>> try to change the meaning of things like "Confirmed vs. Triaged" in your
>> mind.
>>
>> We can probably tag for information we know we need at lot easier. I'm
>> considering something like
>>
>> * needs.system-version
>> * needs.openstack-version
>> * needs.logs
>> * needs.subteam-feedback
>> * has.system-version
>> * has.openstack-version
>> * has.reproduce
>>
>> Some of these a bot can process the text on and tell if that info was
>> provided, and comment how to provide the updated info. Some of this
>> would be human, but with official tags, it would probably help.
>>
>> #3 machine assisted functional tagging
>>
>> I'm playing around with some things that might be useful in mapping new
>> bugs into existing functional buckets like: libvirt, volumes, etc. We'll
>> see how useful it ends up being.
>>
>> #4 reporting on smaller slices
>>
>> Build some tooling to report on the status and change over time of bugs
>> under various tags. This will help visualize how we are doing
>> (hopefully) and where the biggest piles of issues are.
>>
>> The intent is the normal unit of interaction would be one of these
>> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36
>> vmware bugs. It would also highlight the rates of change in these piles,
>> and what's getting attention and what is not.
>>
>>
>> This is going to be kind of an ongoing experiment, but as we currently
>> have no one spear heading bug triage, it seemed like a good time to try
>> this out.
>>
>> Comments and other suggestions are welcomed. The tooling will have the
>> nova flow in mind, but I'm trying to make it so it takes a project name
>> as params on all the scripts, so anyone can use it. It's a little hack
>> and slash right now to discover what the right patterns are.
>>
>> -Sean
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

Re: [openstack-dev] [nova] bug triage experimentation

[Markus Zoeller]

On 28.06.2017 16:50, Sean Dague wrote:
> On 06/28/2017 10:33 AM, Ben Nemec wrote:
>>
>>
>> On 06/23/2017 11:52 AM, Sean Dague wrote:
>>> The Nova bug backlog is just over 800 open bugs, which while
>>> historically not terrible, remains too large to be collectively usable
>>> to figure out where things stand. We've had a few recent issues where we
>>> just happened to discover upgrade bugs filed 4 months ago that needed
>>> fixes and backports.
>>>
>>> Historically we've tried to just solve the bug backlog with volunteers.
>>> We've had many a brave person dive into here, and burn out after 4 - 6
>>> months. And we're currently without a bug lead. Having done a big giant
>>> purge in the past
>>> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
>>>
>>> I know how daunting this all can be.
>>>
>>> I don't think that people can currently solve the bug triage problem at
>>> the current workload that it creates. We've got to reduce the smart
>>> human part of that workload.
>>>
>>> But, I think that we can also learn some lessons from what active github
>>> projects do.
>>>
>>> #1 Bot away bad states
>>>
>>> There are known bad states of bugs - In Progress with no open patch,
>>> Assigned but not In Progress. We can just bot these away with scripts.
>>> Even better would be to react immediately on bugs like those, that helps
>>> to train folks how to use our workflow. I've got some starter scripts
>>> for this up at - https://github.com/sdague/nova-bug-tools
>>
>> Just saw the update on https://bugs.launchpad.net/nova/+bug/1698010 and
>> I don't agree that assigned but not in progress is an invalid state.  If
>> it persists for a period of time then sure, but to me assigning yourself
>> a bug is a signal that you're working on it and nobody else needs to.
>> Otherwise you end up with multiple people working a bug without
>> realizing someone else already was.  I've seen that happen more than once.
> 
> The other case, where folks assign themselves and never do anything,
> happens about 100 times a month.
> 
> We don't live in an exclusive lock environment, anyone can push a fix
> for a bug, and gerrit assigns it to them. I don't see why we'd treat LP
> any differently. Yes, this sometimes leads to duplicate fixes, however
> in the current model it's far more frequent for bugs to be blocked away
> as "assigned" when no one is working on them.
> 
> A future version might be smarter and give folks a 7 day window or
> something, but parsing back the history to understand the right logic
> there is tricky enough that it's a future enhancement at best.
> 
>   -Sean
> 

+1 That happened so frequently I made a query for that:
http://45.55.105.55:8082/bugs-dashboard.html#tabInProgressStale

After poking people to get the actual state, 99% of the time the answer
was "I couldn't work on it, please remove my assignment.".

-- 
Regards, Markus Zoeller (markus_z)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Sean Dague]

On 06/28/2017 10:33 AM, Ben Nemec wrote:
> 
> 
> On 06/23/2017 11:52 AM, Sean Dague wrote:
>> The Nova bug backlog is just over 800 open bugs, which while
>> historically not terrible, remains too large to be collectively usable
>> to figure out where things stand. We've had a few recent issues where we
>> just happened to discover upgrade bugs filed 4 months ago that needed
>> fixes and backports.
>>
>> Historically we've tried to just solve the bug backlog with volunteers.
>> We've had many a brave person dive into here, and burn out after 4 - 6
>> months. And we're currently without a bug lead. Having done a big giant
>> purge in the past
>> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
>>
>> I know how daunting this all can be.
>>
>> I don't think that people can currently solve the bug triage problem at
>> the current workload that it creates. We've got to reduce the smart
>> human part of that workload.
>>
>> But, I think that we can also learn some lessons from what active github
>> projects do.
>>
>> #1 Bot away bad states
>>
>> There are known bad states of bugs - In Progress with no open patch,
>> Assigned but not In Progress. We can just bot these away with scripts.
>> Even better would be to react immediately on bugs like those, that helps
>> to train folks how to use our workflow. I've got some starter scripts
>> for this up at - https://github.com/sdague/nova-bug-tools
> 
> Just saw the update on https://bugs.launchpad.net/nova/+bug/1698010 and
> I don't agree that assigned but not in progress is an invalid state.  If
> it persists for a period of time then sure, but to me assigning yourself
> a bug is a signal that you're working on it and nobody else needs to.
> Otherwise you end up with multiple people working a bug without
> realizing someone else already was.  I've seen that happen more than once.

The other case, where folks assign themselves and never do anything,
happens about 100 times a month.

We don't live in an exclusive lock environment, anyone can push a fix
for a bug, and gerrit assigns it to them. I don't see why we'd treat LP
any differently. Yes, this sometimes leads to duplicate fixes, however
in the current model it's far more frequent for bugs to be blocked away
as "assigned" when no one is working on them.

A future version might be smarter and give folks a 7 day window or
something, but parsing back the history to understand the right logic
there is tricky enough that it's a future enhancement at best.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Ben Nemec]

On 06/23/2017 11:52 AM, Sean Dague wrote:

The Nova bug backlog is just over 800 open bugs, which while
historically not terrible, remains too large to be collectively usable
to figure out where things stand. We've had a few recent issues where we
just happened to discover upgrade bugs filed 4 months ago that needed
fixes and backports.

Historically we've tried to just solve the bug backlog with volunteers.
We've had many a brave person dive into here, and burn out after 4 - 6
months. And we're currently without a bug lead. Having done a big giant
purge in the past
(http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
I know how daunting this all can be.

I don't think that people can currently solve the bug triage problem at
the current workload that it creates. We've got to reduce the smart
human part of that workload.

But, I think that we can also learn some lessons from what active github
projects do.

#1 Bot away bad states

There are known bad states of bugs - In Progress with no open patch,
Assigned but not In Progress. We can just bot these away with scripts.
Even better would be to react immediately on bugs like those, that helps
to train folks how to use our workflow. I've got some starter scripts
for this up at - https://github.com/sdague/nova-bug-tools


Just saw the update on https://bugs.launchpad.net/nova/+bug/1698010 and 
I don't agree that assigned but not in progress is an invalid state.  If 
it persists for a period of time then sure, but to me assigning yourself 
a bug is a signal that you're working on it and nobody else needs to. 
Otherwise you end up with multiple people working a bug without 
realizing someone else already was.  I've seen that happen more than once.


Would it be possible to only un-assign such bugs if they've been in that 
state for a week?  At that point it seems safe to say the assignee has 
either moved on or that the bug is tricky and additional input would be 
useful anyway.


Otherwise, big +1 to a bug bot.  I need to try running it against the 
~700 open TripleO bugs...




#2 Use tag based workflow

One lesson from github projects, is the github tracker has no workflow.
Issues are openned or closed. Workflow has to be invented by every team
based on a set of tags. Sometimes that's annoying, but often times it's
super handy, because it allows the tracker to change workflows and not
try to change the meaning of things like "Confirmed vs. Triaged" in your
mind.

We can probably tag for information we know we need at lot easier. I'm
considering something like

* needs.system-version
* needs.openstack-version
* needs.logs
* needs.subteam-feedback
* has.system-version
* has.openstack-version
* has.reproduce

Some of these a bot can process the text on and tell if that info was
provided, and comment how to provide the updated info. Some of this
would be human, but with official tags, it would probably help.

#3 machine assisted functional tagging

I'm playing around with some things that might be useful in mapping new
bugs into existing functional buckets like: libvirt, volumes, etc. We'll
see how useful it ends up being.

#4 reporting on smaller slices

Build some tooling to report on the status and change over time of bugs
under various tags. This will help visualize how we are doing
(hopefully) and where the biggest piles of issues are.

The intent is the normal unit of interaction would be one of these
smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36
vmware bugs. It would also highlight the rates of change in these piles,
and what's getting attention and what is not.


This is going to be kind of an ongoing experiment, but as we currently
have no one spear heading bug triage, it seemed like a good time to try
this out.

Comments and other suggestions are welcomed. The tooling will have the
nova flow in mind, but I'm trying to make it so it takes a project name
as params on all the scripts, so anyone can use it. It's a little hack
and slash right now to discover what the right patterns are.

-Sean



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Markus Zoeller]

On 26.06.2017 10:49, Sylvain Bauza wrote:
> 
> 
> Le 23/06/2017 18:52, Sean Dague a écrit :
>> The Nova bug backlog is just over 800 open bugs, which while
>> historically not terrible, remains too large to be collectively usable
>> to figure out where things stand. We've had a few recent issues where we
>> just happened to discover upgrade bugs filed 4 months ago that needed
>> fixes and backports.
>>
>> Historically we've tried to just solve the bug backlog with volunteers.
>> We've had many a brave person dive into here, and burn out after 4 - 6
>> months. And we're currently without a bug lead. Having done a big giant
>> purge in the past
>> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
>> I know how daunting this all can be.
>>
>> I don't think that people can currently solve the bug triage problem at
>> the current workload that it creates. We've got to reduce the smart
>> human part of that workload.
>>
> 
> Thanks for sharing ideas, Sean.
> 
>> But, I think that we can also learn some lessons from what active github
>> projects do.
>>
>> #1 Bot away bad states
>>
>> There are known bad states of bugs - In Progress with no open patch,
>> Assigned but not In Progress. We can just bot these away with scripts.
>> Even better would be to react immediately on bugs like those, that helps
>> to train folks how to use our workflow. I've got some starter scripts
>> for this up at - https://github.com/sdague/nova-bug-tools
>>
> 
> Sometimes, I had no idea why but I noticed the Gerrit hook not working
> (ie. amending the Launchpad bug with the Gerrit URL) so some of the bugs
> I was looking for were actively being worked on (and I had the same
> experience myself although my commit msg was pretty correctly marked AFAIR).
> 
> Either way, what you propose sounds reasonable to me. If you care about
> fixing a bug by putting yourself owner of that bug, that also means you
> engage yourself on a resolution sooner than later (even if I do fail
> applying that to myself...).
> 
>> #2 Use tag based workflow
>>
>> One lesson from github projects, is the github tracker has no workflow.
>> Issues are openned or closed. Workflow has to be invented by every team
>> based on a set of tags. Sometimes that's annoying, but often times it's
>> super handy, because it allows the tracker to change workflows and not
>> try to change the meaning of things like "Confirmed vs. Triaged" in your
>> mind.
>>
>> We can probably tag for information we know we need at lot easier. I'm
>> considering something like
>>
>> * needs.system-version
>> * needs.openstack-version
>> * needs.logs
>> * needs.subteam-feedback
>> * has.system-version
>> * has.openstack-version
>> * has.reproduce
>>
>> Some of these a bot can process the text on and tell if that info was
>> provided, and comment how to provide the updated info. Some of this
>> would be human, but with official tags, it would probably help.
>>
> 
> The tags you propose seem to me related to an "Incomplete" vs.
> "Confirmed" state of the bug.
> 
> If I'm not able to triage the bug because I'm missing information like
> the release version or more logs, I put the bug as Incomplete.
> I could add those tags, but I don't see where a programmatical approach
> could help us.
> 
> If I understand correctly, you're rather trying to identify more what's
> missing in the bug report to provide a clear path of resolution, so we
> could mark the bug as Triaged, right? If so, I'd not propose those tags
> for the reason I just said, but rather other tags like (disclaimer, I
> suck at naming things):
> 
>  - rootcause.found
>  - needs.rootcause.analysis
>  - is.regression
>  - reproduced.locally
> 
> 
>> #3 machine assisted functional tagging
>>
>> I'm playing around with some things that might be useful in mapping new
>> bugs into existing functional buckets like: libvirt, volumes, etc. We'll
>> see how useful it ends up being.
>>
> 
> Logs parsing could certainly help. If someone is able to provide a clear
> stacktrace of some root exception, we can get for free the impact
> functional bucket for 80% of cases.
> 
> I'm not fan of identifying a domain by text recognition (like that's not
> because someone tells about libvirt that this is a libvirt bug tho), so
> that's why I'd see more some logs analysis like I mentioned.
> 
> 
>> #4 reporting on smaller slices
>>
>> Build some tooling to report on the status and change over time of bugs
>> under various tags. This will help visualize how we are doing
>> (hopefully) and where the biggest piles of issues are.
>>
>> The intent is the normal unit of interaction would be one of these
>> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36
>> vmware bugs. It would also highlight the rates of change in these piles,
>> and what's getting attention and what is not.
>>
> 
> I do wonder if Markus already wrote such reporting tools. AFAIR, he had
> a couple of very interesting reportings (and he also worked hard on 

Re: [openstack-dev] [nova] bug triage experimentation

[Sean Dague]

On 06/26/2017 04:49 AM, Sylvain Bauza wrote:
> 
> 
> Le 23/06/2017 18:52, Sean Dague a écrit :
>> The Nova bug backlog is just over 800 open bugs, which while
>> historically not terrible, remains too large to be collectively usable
>> to figure out where things stand. We've had a few recent issues where we
>> just happened to discover upgrade bugs filed 4 months ago that needed
>> fixes and backports.
>>
>> Historically we've tried to just solve the bug backlog with volunteers.
>> We've had many a brave person dive into here, and burn out after 4 - 6
>> months. And we're currently without a bug lead. Having done a big giant
>> purge in the past
>> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
>> I know how daunting this all can be.
>>
>> I don't think that people can currently solve the bug triage problem at
>> the current workload that it creates. We've got to reduce the smart
>> human part of that workload.
>>
> 
> Thanks for sharing ideas, Sean.
> 
>> But, I think that we can also learn some lessons from what active github
>> projects do.
>>
>> #1 Bot away bad states
>>
>> There are known bad states of bugs - In Progress with no open patch,
>> Assigned but not In Progress. We can just bot these away with scripts.
>> Even better would be to react immediately on bugs like those, that helps
>> to train folks how to use our workflow. I've got some starter scripts
>> for this up at - https://github.com/sdague/nova-bug-tools
>>
> 
> Sometimes, I had no idea why but I noticed the Gerrit hook not working
> (ie. amending the Launchpad bug with the Gerrit URL) so some of the bugs
> I was looking for were actively being worked on (and I had the same
> experience myself although my commit msg was pretty correctly marked AFAIR).
> 
> Either way, what you propose sounds reasonable to me. If you care about
> fixing a bug by putting yourself owner of that bug, that also means you
> engage yourself on a resolution sooner than later (even if I do fail
> applying that to myself...).
> 
>> #2 Use tag based workflow
>>
>> One lesson from github projects, is the github tracker has no workflow.
>> Issues are openned or closed. Workflow has to be invented by every team
>> based on a set of tags. Sometimes that's annoying, but often times it's
>> super handy, because it allows the tracker to change workflows and not
>> try to change the meaning of things like "Confirmed vs. Triaged" in your
>> mind.
>>
>> We can probably tag for information we know we need at lot easier. I'm
>> considering something like
>>
>> * needs.system-version
>> * needs.openstack-version
>> * needs.logs
>> * needs.subteam-feedback
>> * has.system-version
>> * has.openstack-version
>> * has.reproduce
>>
>> Some of these a bot can process the text on and tell if that info was
>> provided, and comment how to provide the updated info. Some of this
>> would be human, but with official tags, it would probably help.
>>
> 
> The tags you propose seem to me related to an "Incomplete" vs.
> "Confirmed" state of the bug.
> 
> If I'm not able to triage the bug because I'm missing information like
> the release version or more logs, I put the bug as Incomplete.
> I could add those tags, but I don't see where a programmatical approach
> could help us.

We always want that information, and the odds of us getting it from a
user decline over time. When we end up looking at bugs that are year
old, it becomes a big guessing game on their relevancy.

The theory here is that tags like that would be applied by a bot
immediately after the bug is filed. Catching the owner within 5 minutes
of their bug filing with a response which is "we need the following"
means we should get a pretty decent attach rate on that information. And
then you don't have to spend 10 minutes of real human time realizing
that you really need that before moving forward.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] bug triage experimentation

[Sylvain Bauza]

Le 23/06/2017 18:52, Sean Dague a écrit :
> The Nova bug backlog is just over 800 open bugs, which while
> historically not terrible, remains too large to be collectively usable
> to figure out where things stand. We've had a few recent issues where we
> just happened to discover upgrade bugs filed 4 months ago that needed
> fixes and backports.
> 
> Historically we've tried to just solve the bug backlog with volunteers.
> We've had many a brave person dive into here, and burn out after 4 - 6
> months. And we're currently without a bug lead. Having done a big giant
> purge in the past
> (http://lists.openstack.org/pipermail/openstack-dev/2014-September/046517.html)
> I know how daunting this all can be.
> 
> I don't think that people can currently solve the bug triage problem at
> the current workload that it creates. We've got to reduce the smart
> human part of that workload.
> 

Thanks for sharing ideas, Sean.

> But, I think that we can also learn some lessons from what active github
> projects do.
> 
> #1 Bot away bad states
> 
> There are known bad states of bugs - In Progress with no open patch,
> Assigned but not In Progress. We can just bot these away with scripts.
> Even better would be to react immediately on bugs like those, that helps
> to train folks how to use our workflow. I've got some starter scripts
> for this up at - https://github.com/sdague/nova-bug-tools
> 

Sometimes, I had no idea why but I noticed the Gerrit hook not working
(ie. amending the Launchpad bug with the Gerrit URL) so some of the bugs
I was looking for were actively being worked on (and I had the same
experience myself although my commit msg was pretty correctly marked AFAIR).

Either way, what you propose sounds reasonable to me. If you care about
fixing a bug by putting yourself owner of that bug, that also means you
engage yourself on a resolution sooner than later (even if I do fail
applying that to myself...).

> #2 Use tag based workflow
> 
> One lesson from github projects, is the github tracker has no workflow.
> Issues are openned or closed. Workflow has to be invented by every team
> based on a set of tags. Sometimes that's annoying, but often times it's
> super handy, because it allows the tracker to change workflows and not
> try to change the meaning of things like "Confirmed vs. Triaged" in your
> mind.
> 
> We can probably tag for information we know we need at lot easier. I'm
> considering something like
> 
> * needs.system-version
> * needs.openstack-version
> * needs.logs
> * needs.subteam-feedback
> * has.system-version
> * has.openstack-version
> * has.reproduce
> 
> Some of these a bot can process the text on and tell if that info was
> provided, and comment how to provide the updated info. Some of this
> would be human, but with official tags, it would probably help.
> 

The tags you propose seem to me related to an "Incomplete" vs.
"Confirmed" state of the bug.

If I'm not able to triage the bug because I'm missing information like
the release version or more logs, I put the bug as Incomplete.
I could add those tags, but I don't see where a programmatical approach
could help us.

If I understand correctly, you're rather trying to identify more what's
missing in the bug report to provide a clear path of resolution, so we
could mark the bug as Triaged, right? If so, I'd not propose those tags
for the reason I just said, but rather other tags like (disclaimer, I
suck at naming things):

 - rootcause.found
 - needs.rootcause.analysis
 - is.regression
 - reproduced.locally


> #3 machine assisted functional tagging
> 
> I'm playing around with some things that might be useful in mapping new
> bugs into existing functional buckets like: libvirt, volumes, etc. We'll
> see how useful it ends up being.
> 

Logs parsing could certainly help. If someone is able to provide a clear
stacktrace of some root exception, we can get for free the impact
functional bucket for 80% of cases.

I'm not fan of identifying a domain by text recognition (like that's not
because someone tells about libvirt that this is a libvirt bug tho), so
that's why I'd see more some logs analysis like I mentioned.


> #4 reporting on smaller slices
> 
> Build some tooling to report on the status and change over time of bugs
> under various tags. This will help visualize how we are doing
> (hopefully) and where the biggest piles of issues are.
> 
> The intent is the normal unit of interaction would be one of these
> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36
> vmware bugs. It would also highlight the rates of change in these piles,
> and what's getting attention and what is not.
> 

I do wonder if Markus already wrote such reporting tools. AFAIR, he had
a couple of very interesting reportings (and he also worked hard on the
bugs taxonomy) so we could potentially leverage those.

-Sylvain

> 
> This is going to be kind of an ongoing experiment, but as we currently
> have no one spear heading bug triage, it seemed 

Re: [openstack-dev] [nova] bug triage experimentation

[Clay Gerrard]

Sean,

This sounds amazing and Swift could definitely use some [automated]
assistance here.  It would help if you could throw out a WIP somewhere.

First thought that comes to mind tho  storyboard.o.o :\

-Clay

On Fri, Jun 23, 2017 at 9:52 AM, Sean Dague  wrote:

> The Nova bug backlog is just over 800 open bugs, which while
> historically not terrible, remains too large to be collectively usable
> to figure out where things stand. We've had a few recent issues where we
> just happened to discover upgrade bugs filed 4 months ago that needed
> fixes and backports.
>
> Historically we've tried to just solve the bug backlog with volunteers.
> We've had many a brave person dive into here, and burn out after 4 - 6
> months. And we're currently without a bug lead. Having done a big giant
> purge in the past
> (http://lists.openstack.org/pipermail/openstack-dev/2014-
> September/046517.html)
> I know how daunting this all can be.
>
> I don't think that people can currently solve the bug triage problem at
> the current workload that it creates. We've got to reduce the smart
> human part of that workload.
>
> But, I think that we can also learn some lessons from what active github
> projects do.
>
> #1 Bot away bad states
>
> There are known bad states of bugs - In Progress with no open patch,
> Assigned but not In Progress. We can just bot these away with scripts.
> Even better would be to react immediately on bugs like those, that helps
> to train folks how to use our workflow. I've got some starter scripts
> for this up at - https://github.com/sdague/nova-bug-tools
>
> #2 Use tag based workflow
>
> One lesson from github projects, is the github tracker has no workflow.
> Issues are openned or closed. Workflow has to be invented by every team
> based on a set of tags. Sometimes that's annoying, but often times it's
> super handy, because it allows the tracker to change workflows and not
> try to change the meaning of things like "Confirmed vs. Triaged" in your
> mind.
>
> We can probably tag for information we know we need at lot easier. I'm
> considering something like
>
> * needs.system-version
> * needs.openstack-version
> * needs.logs
> * needs.subteam-feedback
> * has.system-version
> * has.openstack-version
> * has.reproduce
>
> Some of these a bot can process the text on and tell if that info was
> provided, and comment how to provide the updated info. Some of this
> would be human, but with official tags, it would probably help.
>
> #3 machine assisted functional tagging
>
> I'm playing around with some things that might be useful in mapping new
> bugs into existing functional buckets like: libvirt, volumes, etc. We'll
> see how useful it ends up being.
>
> #4 reporting on smaller slices
>
> Build some tooling to report on the status and change over time of bugs
> under various tags. This will help visualize how we are doing
> (hopefully) and where the biggest piles of issues are.
>
> The intent is the normal unit of interaction would be one of these
> smaller piles. Be they the 76 libvirt bugs, 61 volumes bugs, or 36
> vmware bugs. It would also highlight the rates of change in these piles,
> and what's getting attention and what is not.
>
>
> This is going to be kind of an ongoing experiment, but as we currently
> have no one spear heading bug triage, it seemed like a good time to try
> this out.
>
> Comments and other suggestions are welcomed. The tooling will have the
> nova flow in mind, but I'm trying to make it so it takes a project name
> as params on all the scripts, so anyone can use it. It's a little hack
> and slash right now to discover what the right patterns are.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev