Re: Reduce bug triage overhead: DjangoAwesomeBot

[Ludvig Ericson]

On May 12, 2009, at 13:24, J. Clifford Dyer wrote:
> I don't see this as a necessary requirement for all patches.  What if
> the patch has nothing but tests in it, and is improving something that
> was sub-optimal or missing altogether in the test code.

I'm sure you can find 99 problems^Wcorner-cases, but I think the vast  
majority of tickets would do well with a "applied, tested and  
confirmed" label.

As I understand it, the bot isn't supposed to replace core  
committers. :-) People will still have to actually look at what they  
commit (gasp), but the bot would give a quick judgement of patch value  
(in terms of "no tests", "tests fail", "patch broken", ...).

No, it won't ever be as fool-proof as everybody wants, but it still  
adds some value, I'd say. The question is though, is the associated  
Trac spamming worth it, and also the issues Russell noted earlier.

I'd say make a proof of concept, talk to a core committer, and then  
decide next move. And as Jacob said, Anybody and Somebody have other  
things to do and always will.

- Ludvig


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reduce bug triage overhead: DjangoAwesomeBot

[J. Clifford Dyer]

On Wed, 2009-05-06 at 18:23 +0800, Russell Keith-Magee wrote:
> 
>  1. Does the patch apply cleanly?
>  2. Does the patch contain a test?
>  3. Does the test fail if the rest of the patch isn't applied?

I don't see this as a necessary requirement for all patches.  What if
the patch has nothing but tests in it, and is improving something that
was sub-optimal or missing altogether in the test code.

>  4. Does it pass when the test is applied?
>  5. Does the patch only solve the surface problem, or does an
> underlying problem exist that needs fixing.
>  6. Does the patch contain documentation?


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reduce bug triage overhead: DjangoAwesomeBot

[Glenn Maynard]

On Wed, May 6, 2009 at 11:39 AM, Henrique Romano  wrote:
> On May 6, 2009, at 7:23 AM, Russell Keith-Magee wrote:
>> 2. Does the patch contain a test?
>> 3. Does the test fail if the rest of the patch isn't applied?

In principle, you can answer these together.  Apply the patch to just
the tests directory, and run tests.  Revert, apply the entire patch,
and run tests again.  In general, everything should pass in the latter
and at least one test should fail in the former.  (Of course, all
tests need to be passing before the patch for this to work, which
isn't always the case.  You could run tests with nothing applied and
ignore the tests that fail, but that's getting hairier, and then it's
running tests three times.)

>> You also need to consider the fact that Trac doesn't much automated
>> metadata to interpret which patches are valid at any given time. A
>> complex ticket may have multiple patches, only some of which will
>> apply at any given time; in addition, there may be competing
>> approaches being discussed, so multiple tickets may be valid.

I'd recommend that it not attempt to triage the entire ticket, but
each individual patch attachment.  Add the results of testing each
patch as a comment: "passed", "test missing", "test failed" with a
backtrace, etc.

Also, standardize on putting the revision number in the patch, eg.
foo-r1234.diff.  The bot can trivially parse this out if it's present,
and test against that revision (else trunk).  This also makes the test
results much more fixed and reproducable.

(Answering the question "does the patch still work today" is useful
too, but would probably need some external interface to request that a
patch be re-tested against trunk.)

-- 
Glenn Maynard

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reduce bug triage overhead: DjangoAwesomeBot

[Henrique Romano]

On May 6, 2009, at 7:23 AM, Russell Keith-Magee wrote:
> Like many things, it sounds like a good idea, but the devil is in the
> details. A tool like this is really only useful if it provides triage
> data that is both accurate and useful. Here's a quick checklist of the
> basic questions a manual triager needs to answer:
>
> 1. Does the patch apply cleanly?
> 2. Does the patch contain a test?
> 3. Does the test fail if the rest of the patch isn't applied?
> 4. Does it pass when the test is applied?
> 5. Does the patch only solve the surface problem, or does an
> underlying problem exist that needs fixing.
> 6. Does the patch contain documentation?
>
> Now, some complications:
> (1) is something that isn't that has an absolute answer - a patch may
> apply to revision X, but may not apply cleanly to revision X+N. This
> point needs to be re-established periodically.

IMO, it will be always a problem (with our without automation)... what  
happens if a patch doesn't apply when it is approved for commit? Do  
you ask for a new patch?


> (2) requires inspection of the patch contents, but that bit isn't too
> hard. However, there are conditions where not having a patch is
> acceptable (things that aren't particularly testable, such as command
> line behaviour), so the non-existence of a test doesn't necessarily
> mean that a patch isn't ready for commit.

Agreed, but the bot should not include the "ready for commit" keyword  
(maybe a "accepted by bot"), since there are others questions to  
consider. About the test itself, I would say that we can use test  
coverage to see if the lines changed were executed by the test suite.

> You also need to consider the fact that Trac doesn't much automated
> metadata to interpret which patches are valid at any given time. A
> complex ticket may have multiple patches, only some of which will
> apply at any given time; in addition, there may be competing
> approaches being discussed, so multiple tickets may be valid.

I think that even tickets being solved by many patches, each patch  
should work alone... but I'm not sure if it is case for django  
development.

> That doesn't mean there isn't a place for a bot like you describe.
> We're all in favor of any tool that will make the triage process
> easier - however, you need to be clear about what the bot will
> contribute to the process before you embark on building a complex
> tool.

Right. I'll write the initial bot work in the project page to validate  
the idea.

--
Henrique


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reduce bug triage overhead: DjangoAwesomeBot

[Almad]

On 6 Kvě, 12:23, Russell Keith-Magee  wrote:
> On Wed, May 6, 2009 at 7:53 AM, Henrique Romano  wrote:
> Like many things, it sounds like a good idea, but the devil is in the
> details. A tool like this is really only useful if it provides triage
> data that is both accurate and useful. Here's a quick checklist of the
> basic questions a manual triager needs to answer:
>
>  1. Does the patch apply cleanly?
>  2. Does the patch contain a test?
>  3. Does the test fail if the rest of the patch isn't applied?
>  4. Does it pass when the test is applied?
>  5. Does the patch only solve the surface problem, or does an
> underlying problem exist that needs fixing.
>  6. Does the patch contain documentation?

Agreed, exactly was I was thinking about.

> Now, some complications:
> (1) is something that isn't that has an absolute answer - a patch may
> apply to revision X, but may not apply cleanly to revision X+N. This
> point needs to be re-established periodically.

Agreed. Well I can think about having it fancier, but for basic I'd
say it's enough to check periodically.

(Pony thought: Database with what files are affected by which patch,
scan timeline every night, check what files were affected by commits,
re-run check for patches that affect those files too)

> (2) requires inspection of the patch contents, but that bit isn't too
> hard. However, there are conditions where not having a patch is
> acceptable (things that aren't particularly testable, such as command
> line behaviour), so the non-existence of a test doesn't necessarily
> mean that a patch isn't ready for commit.

IMO command line is also testable, althrough in an ugly way. Anyway;
we should then create like "test required for this patch" trac
checkbox, or just have "ready for bot" triage state only for more
complicated and tests-requiring patches.

> (3) and (4) require a buildbot to do automatically.

I think about this tool as not a part of buildbot set and I don't
think it require buildbot. This can be done on any machine with
sanitized environment and do not require talking to buildbot.

My idea is a bot that sits in the working directory of django trunk,
applying patches for real, introspecting results by really running
tests and svn st, and reporting results as oomments to the ticket
directly in trac.

> (5) requires human analysis.

Oh I wish I could automate that ;)

> (6) requires inspection of the patch. Again, depending on the ticket,
> non-existence of docs may not be an problem, depending on the ticket.
>
> So - of the 6 tasks, only (1) can be done automatically, and even then
> it needs to be updated regularly. (2), (5) and (6) require human
> evaluation; (3) & (4) require a lot of infrastructure.

AFAIK 3 and 4 require no infrastructure from Django itself and (2) and
(6) could be easily determined if You will have a bot report (patch do
not contain tests & docs and You can see from ticket desc it should be
required).

> You also need to consider the fact that Trac doesn't much automated
> metadata to interpret which patches are valid at any given time. A
> complex ticket may have multiple patches, only some of which will
> apply at any given time; in addition, there may be competing
> approaches being discussed, so multiple tickets may be valid.

For start, I'd say applying latest diff should be enough. For future,
I'd say we want to have special triage state for "ready for bot" and
"bot approved" (easy with trac 0.11 custom workflow) and at this
state, discussion should be resolved.

I don't think it make so much sense for us to try patches that are not
in their final shape.

> That doesn't mean there isn't a place for a bot like you describe.
> We're all in favor of any tool that will make the triage process
> easier - however, you need to be clear about what the bot will
> contribute to the process before you embark on building a complex
> tool.

Like I said, IMHO it can be made a bit easier by some "convention
instead of configuration" decisions.

But yes, still work it is.

> Yours,
> Russ Magee %-)

Almad

P.S.: I just learned Eric has some pieces of this functionality
already and he's intrested in building some functionality during EDC
sprints.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reduce bug triage overhead: DjangoAwesomeBot

[Almad]

On May 6, 1:53 am, Henrique Romano  wrote:
> On May 5, 2009, at 7:03 PM, Jacob Kaplan-Moss wrote:
> It seems a good idea, and IMHO we should try to start implementing  
> something.  I've created a project at google code named django-triage-
> bot, if anyone is interested.

I decided I'd actually like to implement 'triage it for me' part for
trac in general, like

trac-trage 1109

that will download the ticket to my current directory and run the
stuff.

So if someone will implement the 'search for ticket candidates in
django trac' and 'report results', it would be nice.

> Is everybody happy with this triage automation?

That's what I've been originally asking, but as I learned, It's
impossible to judge without proof-of-concept.

> Henrique

Almad

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reduce bug triage overhead: DjangoAwesomeBot

[Henrique Romano]

On May 5, 2009, at 7:03 PM, Jacob Kaplan-Moss wrote:

>
> On Tue, May 5, 2009 at 11:56 PM, Almad  wrote:
>> Cool for me, but have enough pet projects so now, but if anyone have
>> spare weekend to implement it? :)
>
> Unfortunately, Anyone doesn't have a lot of free time these days (nor
> does his brother, Someone).

It seems a good idea, and IMHO we should try to start implementing  
something.  I've created a project at google code named django-triage- 
bot, if anyone is interested.

Is everybody happy with this triage automation?

--
Henrique

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Reduce bug triage overhead: DjangoAwesomeBot

[Jacob Kaplan-Moss]

On Tue, May 5, 2009 at 11:56 PM, Almad  wrote:
> Cool for me, but have enough pet projects so now, but if anyone have
> spare weekend to implement it? :)

Unfortunately, Anyone doesn't have a lot of free time these days (nor
does his brother, Someone).

No, but on a serious note: ideas are cheap; code and time are
expensive. Ideas like these *always* sound good, but it's impossible
to judge without a proof-of-concept. Things don't "just happen" in
open source; they happen when motivated people write them, usually
without anyone's permission or say-so.

Jacob

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Reduce bug triage overhead: DjangoAwesomeBot

[Almad]

Hi,

earlier today, someone was speaking at EuroDjangoCon about how boring
it is to triage tickets (apply against trunk, run tests, etc. etc.)
and that's why it's not progressing smoothly.

I'm thinking: Why not automate this?

Simple bot that will log in to it's trac account, look for "ready for
bot" triage, get ticket, apply tests from the patch against trunk, run
it, make sure it fails, apply whole patch against trunk, make sure
test passed, add a comment, move to "passed ponybot against revision
x'.

And we can get fancy, like check for test coverage in new code and
make sure it's covered. Or that some can post not a final patch, but a
link to git repository with a ticket branch, merge and do above (kinda
more future-resistant than patch).

Generalize it a bit, and it can perhaps be TracAwesomeBot everyone
could benefit from.

Cool for me, but have enough pet projects so now, but if anyone have
spare weekend to implement it? :) Or it could be GSoC '10 project.

Of course, only if it sounds cool for you, that is.

Almad
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---