Re: Stop issue verification?

[Jean Abou Samra]

Le 02/11/2020 à 21:03, Michael Käppler a écrit :

Am 02.11.2020 um 19:47 schrieb Jean Abou Samra:

Le 02/11/2020 à 18:18, Jean Abou Samra a écrit :


Thanks. I'm going ahead to change the labels right now.

Merge request put up at
https://gitlab.com/lilypond/lilypond/-/merge_requests/493
and labels renamed and deleted as appropriate, except for one,
namely Status::started. While verifying what I was doing I
noticed that 51 open issues have this label (as well as,
somewhat surprisingly, 18 closed ones). Most of them
haven't been touched in years; the most recent is from
two months ago though: 
https://gitlab.com/lilypond/lilypond/-/issues/4599

Of course, the wide majority doesn't have an assignee.

List at
https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=opened_name[]=Status%3A%3AStarted 




Is it fine to remove this information? I think it is, but
it seems best to have confirmation as deleting a label
can't be undone (although most of them have were marked
with this label on SourceForge, which makes it appear in
comments).

Hi Jean,
thanks for working on this!
I've had a quick glance on the list of open issues with 
'Status::Started'.


It seems the majority are unfinished patches, where it is often not
clear if the original assignees
did not have time to incorporate review comments or did not want to work
further on the patch at all.
But if anybody does want to work on one of them, he can assign himself
(again).
So I think it is fine to remove 'Status::started', as well.

Cheers,
Michael


Hi all,

So, I did remove Status::Started.

At https://gitlab.com/lilypond/lilypond/-/merge_requests/493there's
some discussion about wether to reopen issues in the event of a
regression. Current CG contains a policy that this should never
be done, always opening a new issue. This dates back to the
Google tracker era, where the work was somehow more organized
than today if I read the archives correctly. I think we don't
need a policy anymore for such a thing: people are perfectly
capable of using their best judgement -- corrected by others
if necessary -- and reopening in some cases definitely makes
sense, such as if the fix was invalidated. What do you think
about removing the following sentence from
http://lilypond.org/doc/latest/Documentation/contributor/issue-classification?

This  means that nobody should ever need look at the report again -- if  there 
is any information in the issue that should be kept, open a  new issue for that 
info.


Thanks,
Jean

Re: Stop issue verification?

[Michael Käppler]

Am 02.11.2020 um 19:47 schrieb Jean Abou Samra:

Le 02/11/2020 à 18:18, Jean Abou Samra a écrit :


Thanks. I'm going ahead to change the labels right now.

Merge request put up at
https://gitlab.com/lilypond/lilypond/-/merge_requests/493
and labels renamed and deleted as appropriate, except for one,
namely Status::started. While verifying what I was doing I
noticed that 51 open issues have this label (as well as,
somewhat surprisingly, 18 closed ones). Most of them
haven't been touched in years; the most recent is from
two months ago though: https://gitlab.com/lilypond/lilypond/-/issues/4599
Of course, the wide majority doesn't have an assignee.

List at
https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=opened_name[]=Status%3A%3AStarted


Is it fine to remove this information? I think it is, but
it seems best to have confirmation as deleting a label
can't be undone (although most of them have were marked
with this label on SourceForge, which makes it appear in
comments).

Hi Jean,
thanks for working on this!
I've had a quick glance on the list of open issues with 'Status::Started'.

It seems the majority are unfinished patches, where it is often not
clear if the original assignees
did not have time to incorporate review comments or did not want to work
further on the patch at all.
But if anybody does want to work on one of them, he can assign himself
(again).
So I think it is fine to remove 'Status::started', as well.

Cheers,
Michael

Re: Stop issue verification?

[Jean Abou Samra]

Le 02/11/2020 à 18:18, Jean Abou Samra a écrit :


Thanks. I'm going ahead to change the labels right now.
Merge request put up at 
https://gitlab.com/lilypond/lilypond/-/merge_requests/493

and labels renamed and deleted as appropriate, except for one,
namely Status::started. While verifying what I was doing I
noticed that 51 open issues have this label (as well as,
somewhat surprisingly, 18 closed ones). Most of them
haven't been touched in years; the most recent is from
two months ago though: https://gitlab.com/lilypond/lilypond/-/issues/4599
Of course, the wide majority doesn't have an assignee.

List at
https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=opened_name[]=Status%3A%3AStarted

Is it fine to remove this information? I think it is, but
it seems best to have confirmation as deleting a label
can't be undone (although most of them have were marked
with this label on SourceForge, which makes it appear in
comments).

Thanks,
Jean

Re: Stop issue verification?

[Jean Abou Samra]

Le 01/11/2020 à 20:50, Dan Eble a écrit :

On Nov 1, 2020, at 14:12, Jean Abou Samra  wrote:

Le 01/11/2020 à 19:11, Jonas Hahnfeld a écrit :


Am Sonntag, den 01.11.2020, 17:03 +0100 schrieb Jean Abou Samra:

In order to finish things properly, I would like to implement
the resolution here.Is this fine by everyone? Jonas?

To recap:
- Update CG,
- Delete the following under "Status": new, accepted, started,
fixed, verified.
- Rename Status::invalid and Status::duplicate into Invalid
and Duplicate, respectively. Since there was no clear
outcome regarding Status::shelved, we could just change it to
Abandonedfor now.

Please don't create confusion between Abandoned and Patch::Abandoned.
If the latter doesn't fit, I'd prefer leaving it as Shelved.

You're right. Let's change Status::Shelved into just Shelved then.

If you're enthusiastic about this, it's fine with me.  Later, we can resume the 
discussion about whether to merge Invalid and Shelved into one tag that covers 
all closed-but-not-fixed issues.
—
Dan

Thanks. I'm going ahead to change the labels right now.

Re: Stop issue verification?

[Dan Eble]

On Nov 1, 2020, at 14:12, Jean Abou Samra  wrote:
> 
> Le 01/11/2020 à 19:11, Jonas Hahnfeld a écrit :
> 
>> Am Sonntag, den 01.11.2020, 17:03 +0100 schrieb Jean Abou Samra:
>>> In order to finish things properly, I would like to implement
>>> the resolution here.Is this fine by everyone? Jonas?
>>> 
>>> To recap:
>>> - Update CG,
>>> - Delete the following under "Status": new, accepted, started,
>>>fixed, verified.
>>> - Rename Status::invalid and Status::duplicate into Invalid
>>>and Duplicate, respectively. Since there was no clear
>>>outcome regarding Status::shelved, we could just change it to
>>>Abandonedfor now.
>> Please don't create confusion between Abandoned and Patch::Abandoned.
>> If the latter doesn't fit, I'd prefer leaving it as Shelved.
> 
> You're right. Let's change Status::Shelved into just Shelved then.

If you're enthusiastic about this, it's fine with me.  Later, we can resume the 
discussion about whether to merge Invalid and Shelved into one tag that covers 
all closed-but-not-fixed issues.
— 
Dan

Re: Stop issue verification?

[Jean Abou Samra]

Le 01/11/2020 à 19:11, Jonas Hahnfeld a écrit :


Am Sonntag, den 01.11.2020, 17:03 +0100 schrieb Jean Abou Samra:

In order to finish things properly, I would like to implement
the resolution here.Is this fine by everyone? Jonas?

To recap:
- Update CG,
- Delete the following under "Status": new, accepted, started,
    fixed, verified.
- Rename Status::invalid and Status::duplicate into Invalid
    and Duplicate, respectively. Since there was no clear
    outcome regarding Status::shelved, we could just change it to
    Abandonedfor now.

Please don't create confusion between Abandoned and Patch::Abandoned.
If the latter doesn't fit, I'd prefer leaving it as Shelved.


You're right. Let's change Status::Shelved into just Shelved then.

Thanks,
Jean

Re: Stop issue verification?

[Jonas Hahnfeld]

Am Sonntag, den 01.11.2020, 17:03 +0100 schrieb Jean Abou Samra:
> In order to finish things properly, I would like to implement
> the resolution here.Is this fine by everyone? Jonas?
> 
> To recap:
> - Update CG,
> - Delete the following under "Status": new, accepted, started,
>    fixed, verified.
> - Rename Status::invalid and Status::duplicate into Invalid
>    and Duplicate, respectively. Since there was no clear
>    outcome regarding Status::shelved, we could just change it to
>    Abandonedfor now.

Please don't create confusion between Abandoned and Patch::Abandoned.
If the latter doesn't fit, I'd prefer leaving it as Shelved.

> 
> I'd like to tackle this tomorrow evening.
> 
> Best regards,
> Jean
> 



signature.asc
Description: This is a digitally signed message part

Re: Stop issue verification?

[Jean Abou Samra]

In order to finish things properly, I would like to implement
the resolution here.Is this fine by everyone? Jonas?

To recap:
- Update CG,
- Delete the following under "Status": new, accepted, started,
  fixed, verified.
- Rename Status::invalid and Status::duplicate into Invalid
  and Duplicate, respectively. Since there was no clear
  outcome regarding Status::shelved, we could just change it to
  Abandonedfor now.

I'd like to tackle this tomorrow evening.

Best regards,
Jean

Re: Stop issue verification?

[Karlin High]

On 10/26/2020 12:42 PM, Jean Abou Samra wrote:
the purpose is to cover also issues that are not actually bugs but were 
opened because of a mis-expectation of the program's behavior 
(currently, "Invalid").


"Not Applicable" or "N/A" or "Does Not Apply"?

"Not A Bug"?

Maybe too negative: "Ignored," "Disregarded,"
--
Karlin High
Missouri, USA

Re: Stop issue verification?

[Jean Abou Samra]

Le 26/10/2020 à 18:32, Karlin High a écrit :


On 10/26/2020 12:16 PM, Jean Abou Samra wrote:
Something like "Void" or "Won't fix" would be a good option in my 
opinion.


Would "Inactive" be too negative or inaccurate?


Not sure about this one: the purpose is to cover also issues that are 
not actually bugs but were opened because of a mis-expectation of the 
program's behavior (currently, "Invalid").

Re: Stop issue verification?

[Karlin High]

On 10/26/2020 12:16 PM, Jean Abou Samra wrote:

Something like "Void" or "Won't fix" would be a good option in my opinion.


Would "Inactive" be too negative or inaccurate?
--
Karlin High
Missouri, USA

Re: Stop issue verification?

[Jean Abou Samra]

Le 26/10/2020 à 17:26, Dan Eble a écrit :


On Oct 26, 2020, at 11:50, Jean Abou Samra  wrote:

   - Status::invalid, Status::duplicate and Status::shelved converted
 to plain labels.

What is the value of "shelved"?  We have 3 issues with that label and I don't 
see what makes them special.


As far as I know, the typical use case would be things like
https://gitlab.com/lilypond/lilypond/-/issues/979
where development moved in completely different directions,
obviating the issue.

I agree that it sounds weird to have it on open issues in
the vein of
https://gitlab.com/lilypond/lilypond/-/issues/2072
On the other hand, there are also many patch-tracking issues
marked "Patch::abandoned" but still open. That's likely a separate
topic. Maybe we could rename Status::shelved to "Abandoned"
instead of "Shelved"?

The main purpose of this tag would be to exclude such tickets from searches of 
open-or-fixed issues, correct?  I don't see a good reason to differentiate 
between Abandoned and Invalid other than possibly to avoid an emotional 
reaction.  If that is a problem, we could possibly work around it by choosing a 
more neutral synonym (Void?) that works for both.
—
Dan


Avoiding emotional reactions is clearly an important part of this 
distinction. Something like "Void" or "Won't fix" would be a good option 
in my opinion.


Best, Jean

Re: Stop issue verification?

[Jean Abou Samra]

Le 26/10/2020 à 17:07, Jonas Hahnfeld a écrit :


Hi,

commenting only on one particular point:

Am Sonntag, den 25.10.2020, 20:43 +0100 schrieb Jean Abou Samra:

In my opinion, the limited total working time of the development
team is better spent on fixing bugs (there are plenty of easy ones
for newbies) than undertaking this task.

I don't think that issue verification requires development knowledge,
quite the opposite as it was meant to be handled by non-programmers.
I do see some benefit in making sure that the original issue is indeed
fixed with the released binaries, but as it would not be me who does
the work, I don't have a strong opinion here.

Jonas


Hi Jonas,

Well, in theory, you are absolutely right. In practice though,
the last rounds of verification were done by Federico, Carl,
Kevin, Trevor, Michael and me, who all qualify at least a bit
as programmers or have other roles in documentation, etc.

To take a concrete example,

https://gitlab.com/lilypond/lilypond/-/issues/644

clearly required basic programming knowledge. I needed
to be able to run lilypond-book from the command line,
and use a script to generate a lytex input file containing
a very large number of small snippets; and verify that
the file I had generated triggered the bug in previous
versions of lilypond-book (since there was no example
on the issue), which in turn required the ability to
manage concurrent installations of LilyPond. None of this
is overly complicated, but as this situation happens
relatively often…

We might decide that this kind of issues ought not
to be verified. Since regression tests are added for
all issues that can be easily demonstrated using a
minimal example, the procedure would come close to
checking the regression test comparison that is
released. And that can also be handled by non-programmers…

Cheers,
Jean

Re: Stop issue verification?

[Dan Eble]

On Oct 26, 2020, at 11:50, Jean Abou Samra  wrote:
>>>   - Status::invalid, Status::duplicate and Status::shelved converted
>>> to plain labels.
>> What is the value of "shelved"?  We have 3 issues with that label and I 
>> don't see what makes them special.
> 
> 
> As far as I know, the typical use case would be things like
> https://gitlab.com/lilypond/lilypond/-/issues/979
> where development moved in completely different directions,
> obviating the issue.
> 
> I agree that it sounds weird to have it on open issues in
> the vein of
> https://gitlab.com/lilypond/lilypond/-/issues/2072
> On the other hand, there are also many patch-tracking issues
> marked "Patch::abandoned" but still open. That's likely a separate
> topic. Maybe we could rename Status::shelved to "Abandoned"
> instead of "Shelved"?

The main purpose of this tag would be to exclude such tickets from searches of 
open-or-fixed issues, correct?  I don't see a good reason to differentiate 
between Abandoned and Invalid other than possibly to avoid an emotional 
reaction.  If that is a problem, we could possibly work around it by choosing a 
more neutral synonym (Void?) that works for both.
— 
Dan

Re: Stop issue verification?

[Jonas Hahnfeld]

Hi,

commenting only on one particular point:

Am Sonntag, den 25.10.2020, 20:43 +0100 schrieb Jean Abou Samra:
> In my opinion, the limited total working time of the development
> team is better spent on fixing bugs (there are plenty of easy ones
> for newbies) than undertaking this task.

I don't think that issue verification requires development knowledge,
quite the opposite as it was meant to be handled by non-programmers.
I do see some benefit in making sure that the original issue is indeed
fixed with the released binaries, but as it would not be me who does
the work, I don't have a strong opinion here.

Jonas


signature.asc
Description: This is a digitally signed message part

Re: Stop issue verification?

[Jean Abou Samra]

Le 26/10/2020 à 13:14, Dan Eble a écrit :

On Oct 25, 2020, at 15:43, Jean Abou Samra  wrote:

I propose that we:

- Remove this procedure.

- Also remove the "Status" scoped label, adding non-scoped labels
   when necessary.

   - Status::new and Status::accepted: no longer applicable since
 there is no longer a formal bug squad.

I'm pretty sure I can support this in any case.



   - Status::fixed and Status::verified: not needed anymore if there
 is no verification (closed issues are fixed).

I guess you mean "closed issues that are not tagged as invalid are fixed."



Indeed. The idea is that this case is more frequent than Invalid
or Duplicate, and thus doesn't deserve any special label.



   - Status::started: assign the issue to yourself if you are working
 on it. This appears in the UI and you also see if an issue has
 an associated merge request.

Agreed.



   - Status::invalid, Status::duplicate and Status::shelved converted
 to plain labels.

What is the value of "shelved"?  We have 3 issues with that label and I don't 
see what makes them special.



As far as I know, the typical use case would be things like
https://gitlab.com/lilypond/lilypond/-/issues/979
where development moved in completely different directions,
obviating the issue.

I agree that it sounds weird to have it on open issues in
the vein of
https://gitlab.com/lilypond/lilypond/-/issues/2072
On the other hand, there are also many patch-tracking issues
marked "Patch::abandoned" but still open. That's likely a separate
topic. Maybe we could rename Status::shelved to "Abandoned"
instead of "Shelved"?



- Explicitely state in [CG 9.1] that whenever applicable, bug fixes
   should contain a regression test − currently it says this needs a policy.

I would not make it a policy, but I would say that if you want your 
contribution to work six months from now, you should make an effort to cover it 
in regression tests.



Agreed.

Best,
Jean

Re: Stop issue verification?

[Dan Eble]

On Oct 25, 2020, at 15:43, Jean Abou Samra  wrote:
> 
> I propose that we:
> 
> - Remove this procedure.
> 
> - Also remove the "Status" scoped label, adding non-scoped labels
>   when necessary.
> 
>   - Status::new and Status::accepted: no longer applicable since
> there is no longer a formal bug squad.

I'm pretty sure I can support this in any case.


>   - Status::fixed and Status::verified: not needed anymore if there
> is no verification (closed issues are fixed).

I guess you mean "closed issues that are not tagged as invalid are fixed."


>   - Status::started: assign the issue to yourself if you are working
> on it. This appears in the UI and you also see if an issue has
> an associated merge request.

Agreed.


>   - Status::invalid, Status::duplicate and Status::shelved converted
> to plain labels.

What is the value of "shelved"?  We have 3 issues with that label and I don't 
see what makes them special.


> - Explicitely state in [CG 9.1] that whenever applicable, bug fixes
>   should contain a regression test − currently it says this needs a policy.

I would not make it a policy, but I would say that if you want your 
contribution to work six months from now, you should make an effort to cover it 
in regression tests.
— 
Dan

Stop issue verification?

[Jean Abou Samra]

Hi all,

Before 2.21.80 is released, I'd like to restart this debate
(which might be contentious; we'll see).

Basically, I think our current processes obviate the effort
to go through all fixed issues and verify them. The primary job
was to ensure that patches had landed in master, due to a largely
hand-powered workflow that could be error-prone. In my experience,
many of the issues cannot be verified easily (especially when
involving the build system, Guile 2, etc.), and many others are
not actually bugs but documentation wishes for example. This all
takes a good amount of time, with little actual benefits as far
as I could see.The bugs for which verification is workable are
exactly those for which we are adding regression tests.

In my opinion, the limited total working time of the development
team is better spent on fixing bugs (there are plenty of easy ones
for newbies) than undertaking this task.

I propose that we:

- Remove this procedure.

- Also remove the "Status" scoped label, adding non-scoped labels
  when necessary.

  - Status::new and Status::accepted: no longer applicable since
    there is no longer a formal bug squad.

  - Status::fixed and Status::verified: not needed anymore if there
    is no verification (closed issues are fixed).

  - Status::started: assign the issue to yourself if you are working
    on it. This appears in the UI and you also see if an issue has
    an associated merge request.

  - Status::invalid, Status::duplicate and Status::shelved converted
    to plain labels.

- Explicitely state in [CG 9.1] that whenever applicable, bug fixes
  should contain a regression test − currently it says this needs a policy.

What do you think?

Regards,
Jean

[CG 9.1] 
http://lilypond.org/doc/v2.21/Documentation/contributor/introduction-to-regression-tests

Re: issue verification

[Jean Abou Samra]

Hi,

Le 21/09/2020 à 14:30, Michael Käppler a écrit :

Am 18.09.2020 um 23:56 schrieb Jean Abou Samra:

Hi,

Le 18/09/2020 à 11:15, Phil Holmes a écrit :


I don't know if this isn't clear, but just to state the original
point of verifying issues.

If the change is claimed to fix a bug, we compile the previously
buggy code with the release in which the bug is claimed fixed and
check that the bug is no longer there.  It it has really disappeared,
we change the status of the issue from Fixed to Verified.  i.e. we
are certain that the bug is no longer there.

If the change is to provide updated functionality, then it can be
really quite hard to verify the new functionality, and in any case
the patch review system should do that.  So we simply check the patch
was pushed into the claimed build.  If it's clear that it was, we
mark the status as Verified.

That was the original intention of verifying issues.


Thanks for the insight.

So, Michael and I verified nearly all issues in Status::Fixed.

Actually, Jean did the vast majority, because I had to leave on Friday
afternoon for a rehearsal weekend.


Don't worry, the majority of them was actually tracking patches,
and those were pretty quickly verified.



The ones remaining are listed here:

https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=closed_name[]=Status%3A%3AFixed[milestone_title]=2.21.7 




Among these,

https://gitlab.com/lilypond/lilypond/-/issues/6027

which I don't know how to test (Jonas?).

The others (5 items) are Windows-related. Maybe Michael or someone
else with access to a Windows machine can go over them?

I verified them and noticed a couple of another issues.
(#6044-#6046).


Thanks!



Lilypond-book on Windows seems broken to me, because
the correct way to run it is neither obvious nor documented.
The script lacks a file ending (2.20.0 had lilypond-book.py, 2.21.6 only
lilypond-book) and can only be run like 'python lilypond-book foo.lytex'

Jean, what would you propose as a good way to ship these python scripts
for the Windows releases?


I'm not knowledgeable about this, sorry.You'd have to ask Jonas for example.




Otherwise, part of the work was extremely simple, namely check that
the commits had made it into some branch under release/, which is fast;
and part of it was much harder, when trying to reproduce memory-related
bugs, for example. I reopened a few of these issues.

This doesn't answer Jonas' initial question: what should be done,
in terms of policies, with regard to issue verification? I don't
know. Personally I think it's not really needed as long as every
bug fix contains a regression test.

Do we agree that 'verification' in the sense of
'checking whether a particular commit made it into a release x.y.z'
is not necessary anymore, because attached merge requests show that
the commit has gone in?


I think there is sufficient agreement on that. Do you want to prepare
a merge request to update the CG?


Speaking of regression tests, I also went through comparison pages
from "2.21.6 vs. 2.21.5" to "2.21.2 vs. 2.21.1" included. Honestly,
I've had my share of this; could somebody jump in to verify the
last two pages as well? These are:

lilypond.org/test/v2.21.1-1/compare-v2.21.0-1/index.html

http://lilypond.org/test/v2.21.0-1/compare-v2.20.0-1/index.html

I find that very difficult, because some changes may be intended, others
not...
Also there are often slight spacing changes that are probably 
"Heisenbugs",

like James mentioned.


Those are the tricky part, indeed.

Best,
Jean

Re: issue verification

[Michael Käppler]

Am 18.09.2020 um 23:56 schrieb Jean Abou Samra:

Hi,

Le 18/09/2020 à 11:15, Phil Holmes a écrit :


I don't know if this isn't clear, but just to state the original
point of verifying issues.

If the change is claimed to fix a bug, we compile the previously
buggy code with the release in which the bug is claimed fixed and
check that the bug is no longer there.  It it has really disappeared,
we change the status of the issue from Fixed to Verified.  i.e. we
are certain that the bug is no longer there.

If the change is to provide updated functionality, then it can be
really quite hard to verify the new functionality, and in any case
the patch review system should do that.  So we simply check the patch
was pushed into the claimed build.  If it's clear that it was, we
mark the status as Verified.

That was the original intention of verifying issues.


Thanks for the insight.

So, Michael and I verified nearly all issues in Status::Fixed.

Actually, Jean did the vast majority, because I had to leave on Friday
afternoon for a rehearsal weekend.


The ones remaining are listed here:

https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=closed_name[]=Status%3A%3AFixed[milestone_title]=2.21.7


Among these,

https://gitlab.com/lilypond/lilypond/-/issues/6027

which I don't know how to test (Jonas?).

The others (5 items) are Windows-related. Maybe Michael or someone
else with access to a Windows machine can go over them?

I verified them and noticed a couple of another issues.
(#6044-#6046)

Lilypond-book on Windows seems broken to me, because
the correct way to run it is neither obvious nor documented.
The script lacks a file ending (2.20.0 had lilypond-book.py, 2.21.6 only
lilypond-book) and can only be run like 'python lilypond-book foo.lytex'

Jean, what would you propose as a good way to ship these python scripts
for the Windows releases?



Otherwise, part of the work was extremely simple, namely check that
the commits had made it into some branch under release/, which is fast;
and part of it was much harder, when trying to reproduce memory-related
bugs, for example. I reopened a few of these issues.

This doesn't answer Jonas' initial question: what should be done,
in terms of policies, with regard to issue verification? I don't
know. Personally I think it's not really needed as long as every
bug fix contains a regression test.

Do we agree that 'verification' in the sense of
'checking whether a particular commit made it into a release x.y.z'
is not necessary anymore, because attached merge requests show that
the commit has gone in?



Speaking of regression tests, I also went through comparison pages
from "2.21.6 vs. 2.21.5" to "2.21.2 vs. 2.21.1" included. Honestly,
I've had my share of this; could somebody jump in to verify the
last two pages as well? These are:

lilypond.org/test/v2.21.1-1/compare-v2.21.0-1/index.html

http://lilypond.org/test/v2.21.0-1/compare-v2.20.0-1/index.html

I find that very difficult, because some changes may be intended, others
not...
Also there are often slight spacing changes that are probably "Heisenbugs",
like James mentioned.


Depending on your time and energy available, you may even go
farther. Or simply (I mean simple, not costless) verify the
entire test suite:

http://lilypond.org/doc/v2.21/input/regression/collated-files.html

That'll be all for today.

Thank you very much, Jean!


Best,
Jean

Re: issue verification

[Jean Abou Samra]

Hi,

Le 18/09/2020 à 11:15, Phil Holmes a écrit :

I don't know if this isn't clear, but just to state the original point 
of verifying issues.


If the change is claimed to fix a bug, we compile the previously buggy 
code with the release in which the bug is claimed fixed and check that 
the bug is no longer there.  It it has really disappeared, we change 
the status of the issue from Fixed to Verified.  i.e. we are certain 
that the bug is no longer there.


If the change is to provide updated functionality, then it can be 
really quite hard to verify the new functionality, and in any case the 
patch review system should do that.  So we simply check the patch was 
pushed into the claimed build.  If it's clear that it was, we mark the 
status as Verified.


That was the original intention of verifying issues.


Thanks for the insight.

So, Michael and I verified nearly all issues in Status::Fixed.
The ones remaining are listed here:

https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=closed_name[]=Status%3A%3AFixed[milestone_title]=2.21.7

Among these,

https://gitlab.com/lilypond/lilypond/-/issues/6027

which I don't know how to test (Jonas?).

The others (5 items) are Windows-related. Maybe Michael or someone
else with access to a Windows machine can go over them?

Otherwise, part of the work was extremely simple, namely check that
the commits had made it into some branch under release/, which is fast;
and part of it was much harder, when trying to reproduce memory-related
bugs, for example. I reopened a few of these issues.

This doesn't answer Jonas' initial question: what should be done,
in terms of policies, with regard to issue verification? I don't
know. Personally I think it's not really needed as long as every
bug fix contains a regression test.

Speaking of regression tests, I also went through comparison pages
from "2.21.6 vs. 2.21.5" to "2.21.2 vs. 2.21.1" included. Honestly,
I've had my share of this; could somebody jump in to verify the
last two pages as well? These are:

lilypond.org/test/v2.21.1-1/compare-v2.21.0-1/index.html

http://lilypond.org/test/v2.21.0-1/compare-v2.20.0-1/index.html

Depending on your time and energy available, you may even go
farther. Or simply (I mean simple, not costless) verify the
entire test suite:

http://lilypond.org/doc/v2.21/input/regression/collated-files.html

That'll be all for today.

Best,
Jean

Re: issue verification

[Phil Holmes]

I don't know if this isn't clear, but just to state the original point of 
verifying issues.


If the change is claimed to fix a bug, we compile the previously buggy code 
with the release in which the bug is claimed fixed and check that the bug is 
no longer there.  It it has really disappeared, we change the status of the 
issue from Fixed to Verified.  i.e. we are certain that the bug is no longer 
there.


If the change is to provide updated functionality, then it can be really 
quite hard to verify the new functionality, and in any case the patch review 
system should do that.  So we simply check the patch was pushed into the 
claimed build.  If it's clear that it was, we mark the status as Verified.


That was the original intention of verifying issues.

--
Phil Holmes


- Original Message - 
From: "Jonas Hahnfeld" 
To: "Michael Käppler" ; "Jean Abou Samra" 
; "lilypond-devel" 

Sent: Friday, September 18, 2020 7:57 AM
Subject: Re: issue verification

Re: issue verification

[Federico Bruni]

Il giorno ven 18 set 2020 alle 14:48, Michael Käppler 
 ha scritto:

Having looked at only four issues now, there were already two that
described bugs, but lacked a description how to reproduce the problem


Yes, it's quite common: I would say >90% of cases.
In my experience verifying issues was mainly checking the presence of 
the commit in a git tag.

Current Gitlab workflow makes this kind of verification obsolete.

Re: issue verification

[Michael Käppler]

Am 18.09.2020 um 13:07 schrieb Michael Käppler:

Am 18.09.2020 um 11:15 schrieb Phil Holmes:

I don't know if this isn't clear, but just to state the original point
of verifying issues.

If the change is claimed to fix a bug, we compile the previously buggy
code with the release in which the bug is claimed fixed and check that
the bug is no longer there.  It it has really disappeared, we change
the status of the issue from Fixed to Verified.  i.e. we are certain
that the bug is no longer there.

If the change is to provide updated functionality, then it can be
really quite hard to verify the new functionality, and in any case the
patch review system should do that.  So we simply check the patch was
pushed into the claimed build.  If it's clear that it was, we mark the
status as Verified.

That was the original intention of verifying issues.

Thank you, Phil,
that confirms my previous thoughts.

I can start with verifying issues for 2.21.2 today.

Just started, some further thoughts:

1. Having looked at only four issues now, there were already two that
described bugs, but lacked a description how to reproduce the problem.
See #6001 with a link to a very long thread on lilypond-devel and
#5995 with no clear error description. ('clear' in the sense of
understandable
for people that want to help verifying fixes and are not necessarily
involved with
coding.

2. Some issues cannot be verified with the actual releases since they
refer to special functionality like the GS API, which is not active in
the releases.  (See #5996)

3. Some issues can only be verified with special tools
(See #5983)

Cheers,
Michael

Re: issue verification

[Michael Käppler]

Am 18.09.2020 um 11:15 schrieb Phil Holmes:

I don't know if this isn't clear, but just to state the original point
of verifying issues.

If the change is claimed to fix a bug, we compile the previously buggy
code with the release in which the bug is claimed fixed and check that
the bug is no longer there.  It it has really disappeared, we change
the status of the issue from Fixed to Verified.  i.e. we are certain
that the bug is no longer there.

If the change is to provide updated functionality, then it can be
really quite hard to verify the new functionality, and in any case the
patch review system should do that.  So we simply check the patch was
pushed into the claimed build.  If it's clear that it was, we mark the
status as Verified.

That was the original intention of verifying issues.

Thank you, Phil,
that confirms my previous thoughts.

I can start with verifying issues for 2.21.2 today.

Cheers,
Michael



--
Phil Holmes


- Original Message - From: "Jonas Hahnfeld" 
To: "Michael Käppler" ; "Jean Abou Samra"
; "lilypond-devel" 
Sent: Friday, September 18, 2020 7:57 AM
Subject: Re: issue verification

Re: issue verification

[Michael Käppler]

Am 18.09.2020 um 08:57 schrieb Jonas Hahnfeld:

Am Donnerstag, den 17.09.2020, 23:01 +0200 schrieb Michael Käppler:

 From my point of view, if an issue is
1. closed
2. marked as "Status::Fixed"
3. assigned to milestone x.y.z

one can be sure that it is fixed in release x.y.z
What should we verify?

See below.


Another point is what Jean mentioned, 'verifying' in the sense of
checking, whether a particular issue is really fixed in the release
that corresponds to the issue's milestone.

Yes, that is what CG mentions should be done, please refer there.

I found the following section ambiguous:



"If the issue refers to a bug, try to reproduce the bug with the latest
officially released version (not one you’ve built yourself from source);
if the bug is no longer there, mark the issue “Status::Verified” (i.e.,
“the fix has been verified to work”).

Quite a few of these will be issues tracking patches. *You do not have
to prove these patches work - simply that they have been pushed into the
code base.* The developer should have put information similar to “Pushed
as as d8fce1e1ea2aca1a82e25e47805aef0f70f511b9” in the tracker. The long
list of letters and numbers is called the “committish”."



Please let me give another try:

1. If an issue refers to a bug and is marked as Status::Fixed, test if
the bug is really gone in the latest release.
2. If an issue refers to another change (Enhancement, Documentation,
...) and has a MR assigned, verify if the corresponding MR
has gone into the release. (This should not be necessary with the
current process, IMHO).


Do you mean we should try to start a new community effort to do that?

I'm not implying anything. The question was that somebody who knows
about the procedure (not me) should go over the instructions given in
the documentation. Until that happens, it's pointless to discuss among
three people (you, Jean, me) who don't know about it.

Ok. My starting point was to help with the verifying process, which is
only possible, if I
understand what we want to do and how we want it to be done.
I'm not sure if waiting for more insightful people is promising, given
that up to now
nobody else replied.
Maybe the discussion gives some insight where the CG could be clearer
for people
like me who do not know about the procedure.

Thanks for your time,
Michael

Re: issue verification

[Jonas Hahnfeld]

Am Donnerstag, den 17.09.2020, 23:01 +0200 schrieb Michael Käppler:
> From my point of view, if an issue is
> 1. closed
> 2. marked as "Status::Fixed"
> 3. assigned to milestone x.y.z
> 
> one can be sure that it is fixed in release x.y.z
> What should we verify?

See below.

> Another point is what Jean mentioned, 'verifying' in the sense of
> checking, whether a particular issue is really fixed in the release
> that corresponds to the issue's milestone.

Yes, that is what CG mentions should be done, please refer there.

> Do you mean we should try to start a new community effort to do that?

I'm not implying anything. The question was that somebody who knows
about the procedure (not me) should go over the instructions given in
the documentation. Until that happens, it's pointless to discuss among
three people (you, Jean, me) who don't know about it.

Jonas


signature.asc
Description: This is a digitally signed message part

Re: issue verification

[Michael Käppler]

Am 17.09.2020 um 23:01 schrieb Michael Käppler:

From my point of view, if an issue is
1. closed
2. marked as "Status::Fixed"
3. assigned to milestone x.y.z

one can be sure that it is fixed in release x.y.z

What I meant was: 'that the commit with the purpose of
fixing the issue has gone into release x.y.z'

Re: issue verification

[Michael Käppler]

Am 17.09.2020 um 21:32 schrieb Jonas Hahnfeld:

Am Donnerstag, den 17.09.2020, 20:58 +0200 schrieb Jean Abou Samra:

What I do not yet understand is when milestones should be assigned
to issues and MRs. So far Jonas has done this from time to time. That was
after each release, right?

Yes, mostly so. I'm just continuing what CG previously said about the
Fixed_x.y.z labels: Closed issues should be labeled Status::Fixed and
assigned a label, now a milestone, so they can be found. Additionally,
I assign all merge requests without a milestone after Phil does a
release and I need to create a new milestone anyway. That has the
benefit that the milestone bundles all information about the changes
that went into a release, in a single place.´

I still do not understand why this verification could be necessary in the
current process:

1. Phil did a release, let's say 2.21.7. you open a new milestone for
the next release,
2.21.8. You assign all merge requests that do not have a milestone (that
is, all
that were merged between 2.21.6 and 2.21.7) to milestone 2.21.7.
2. Someone tackles an open issue with a merge request. He writes
'Closes #1234' in the description. After the MR has been accepted and
merged, issue #1234 will be automatically closed.
(But Status::Fixed must be set manually, right?)
3. Phil does 2.21.8, and you or someone else does assign the milestone
'2.21.8'
to closed and fixed issues.

The cycle starts from the beginning.

From my point of view, if an issue is
1. closed
2. marked as "Status::Fixed"
3. assigned to milestone x.y.z

one can be sure that it is fixed in release x.y.z
What should we verify?

Another point is what Jean mentioned, 'verifying' in the sense of
checking, whether a particular issue is really fixed in the release
that corresponds to the issue's milestone.

Do you mean we should try to start a new community effort to do that?

Michael

Re: issue verification

[Jean Abou Samra]

Le 17/09/2020 à 22:34, Jonas Hahnfeld a écrit :

Am Donnerstag, den 17.09.2020, 20:30 + schrieb Trevor:

Maybe I'm misunderstanding the problem, but a list of issues to verify
for a particular release, say 2.21.4, can be found by

Issues -> Milestones -> Release LilyPond 2.21.4 -> Milestone 2.21.4
Page down to see a list of completed issues (closed).

These are the ones to verify for 2.21.4 I believe.

Right, that's what we currently have. As far as I understood Jean
wondered if we could move the step of assigning the milestone to
verification? In that case, the described procedure would not work.


Yes, this answers my question.

In the future I'll assign milestones when merging my MRs and closing issues.

Best,
Jean

Re: Re[2]: issue verification

[Jonas Hahnfeld]

Am Donnerstag, den 17.09.2020, 20:30 + schrieb Trevor:
> Jonas, you wrote 17/09/2020 20:32:31
> Subject: Re: issue verification
> 
> > The problem I'm seeing is that it would become harder to find the
> > issues to verify: Most of the time, there will be fixed issues for the
> > current version still in development, which cannot be verified. Making
> > everybody wanting to help with verification skip over the same set of
> > issues potentially wastes a lot of time...
> > 
> > Jonas
> Maybe I'm misunderstanding the problem, but a list of issues to verify
> for a particular release, say 2.21.4, can be found by
> 
> Issues -> Milestones -> Release LilyPond 2.21.4 -> Milestone 2.21.4
> Page down to see a list of completed issues (closed).
> 
> These are the ones to verify for 2.21.4 I believe.

Right, that's what we currently have. As far as I understood Jean
wondered if we could move the step of assigning the milestone to
verification? In that case, the described procedure would not work.


signature.asc
Description: This is a digitally signed message part

Re[2]: issue verification

[Trevor]

Jonas, you wrote 17/09/2020 20:32:31
Subject: Re: issue verification


The problem I'm seeing is that it would become harder to find the
issues to verify: Most of the time, there will be fixed issues for the
current version still in development, which cannot be verified. Making
everybody wanting to help with verification skip over the same set of
issues potentially wastes a lot of time...

Jonas

Maybe I'm misunderstanding the problem, but a list of issues to verify
for a particular release, say 2.21.4, can be found by

Issues -> Milestones -> Release LilyPond 2.21.4 -> Milestone 2.21.4
Page down to see a list of completed issues (closed).

These are the ones to verify for 2.21.4 I believe.

Trevor

Re: issue verification

[Jonas Hahnfeld]

Am Donnerstag, den 17.09.2020, 20:58 +0200 schrieb Jean Abou Samra:
> What I do not yet understand is when milestones should be assigned
> to issues and MRs. So far Jonas has done this from time to time. That was
> after each release, right?

Yes, mostly so. I'm just continuing what CG previously said about the
Fixed_x.y.z labels: Closed issues should be labeled Status::Fixed and
assigned a label, now a milestone, so they can be found. Additionally,
I assign all merge requests without a milestone after Phil does a
release and I need to create a new milestone anyway. That has the
benefit that the milestone bundles all information about the changes
that went into a release, in a single place.

> Should we switch to doing this as part of the verification process?

The problem I'm seeing is that it would become harder to find the
issues to verify: Most of the time, there will be fixed issues for the
current version still in development, which cannot be verified. Making
everybody wanting to help with verification skip over the same set of
issues potentially wastes a lot of time...

Jonas


signature.asc
Description: This is a digitally signed message part

Re: issue verification

[Jean Abou Samra]

Hi,

Le 17/09/2020 à 13:27, Michael Käppler a écrit :


Am 17.09.2020 um 09:47 schrieb Jonas Hahnfeld:


Hi all,

unless I'm missing something, there has been no activity on this since
the community effort in May [1][2] and issues since version 2.21.2
don't yet have Status::Verified labels. As I'm currently updating CG to
mention our use of milestones (not sure why I forgot about this
previously), the verification procedure should probably also get
updated.
I haven't been active in this area (and don't plan to be), so it would
be great if somebody could look over the current description (assuming
the procedure should be continued).

To give a head start, here's again the list of all Status::Fixed:
https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=closed_name[]=Status%3A%3AFixed 



Furthermore you can directly get to the issues associated with a
milestone, for example for 2.21.2:
https://gitlab.com/lilypond/lilypond/-/milestones/320#tab-labels

Thanks,
Jonas

1: 
https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00284.html
2: 
https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00379.html

Hi Jonas,
I just took a quick glance on it and have many questions... :)
Let me summarize them with an example:

Take
https://gitlab.com/lilypond/lilypond/-/issues/5999

1. Federico wrote in the thread you mentioned [1]:
"In the last comment you should find a commit id (if it's missing you'll
have to search it)."

I think this refers to the time before Gitlab, when issues had
a message like

Trim unused toplevel targets.
author    Han-Wen Nienhuys 
    Sun, 22 Mar 2020 00:16:36 +0100 (00:16 +0100)
committer    Han-Wen Nienhuys 
    Tue, 31 Mar 2020 21:18:36 +0100 (22:18 +0200)
commit    0755e2f3dbb99ac256fb447d6ed14d3fe87b8ab5

at the end of the thread, right?


I think so.



2. So I thought I had to look at the corresponding MRs to find
the commits I shall verify.
For #5999, there are two MRs listed as 'Related', which are:

https://gitlab.com/lilypond/lilypond/-/merge_requests/145
Add PDF bookmark support & multi-level TOC outline ability

and

https://gitlab.com/lilypond/lilypond/-/merge_requests/147
Avoid scm_append_x for property values

I'm not sure how !145 is related to #5999. Maybe #5999 did show up
after !145, and !147 did fix it.

Anyway, am I right that I have to extract all commit ids from both MRs
and check if they got into 2.21.2? Or do I have some faulty thinking 
here?

This would be very complicated and time-consuming, especially when MRs
consist of several commits.

3. What I do not get: A milestone is simply a named period of time, to 
which

you can assign MRs and issues, right?



As far as I understand, that's it, roughly. Milestones are just a kind 
of extended
label concept (I think they integrate with the release feature somehow, 
too). At any

rate, the documentation is over there:
https://docs.gitlab.com/ee/user/project/milestones/


So after each release, you (or
someone else)
start a new milestone with the next version number. If an issue has been
closed, you
set the issue's milestone to the current milestone.
An issue is closed automatically when the corresponding MR has "Closes
#..." in the description.
So if an issue is closed and the milestone of the next release is added,
what is there left
to verify? Can't we be sure that the commits of the corresponding,
closed MR have gone in?

Michael


In my opinion, we no longer need to verify patches. That is, there is
no need to check that the commits are present in master using committishes.
This was done when branches were manually managed, meaning that a patch 
could

have slipped through because of a wrong usage of Git; I hardly see how
something could go wrong on GitLab.

Basically, the new setup for me would mean installing the development
release (downloaded from lilypond.org, not self-compiled, because
build issues are a non-negligible source for problems -- the Contributor's
Guide prescribes this), and going through all Status::Fixed issues,
testing wether the bug is effectively gone. And if there is a need for a
regression test or more documentation, opening a new issue for this.

What I do not yet understand is when milestones should be assigned
to issues and MRs. So far Jonas has done this from time to time. That was
after each release, right? Should we switch to doing this as part of the
verification process?

By the way, verifying regression tests is also in order…

Thoughts? I can help a bit next week-end.

For reference, here's the relevant CG chapter, which is to be updated:

http://lilypond.org/doc/v2.21/Documentation/contributor/bug-squad-checklists

Thanks,
Jean

Re: issue verification

[Michael Käppler]

Am 17.09.2020 um 09:47 schrieb Jonas Hahnfeld:


Hi all,

unless I'm missing something, there has been no activity on this since
the community effort in May [1][2] and issues since version 2.21.2
don't yet have Status::Verified labels. As I'm currently updating CG to
mention our use of milestones (not sure why I forgot about this
previously), the verification procedure should probably also get
updated.
I haven't been active in this area (and don't plan to be), so it would
be great if somebody could look over the current description (assuming
the procedure should be continued).

To give a head start, here's again the list of all Status::Fixed:
https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=closed_name[]=Status%3A%3AFixed

Furthermore you can directly get to the issues associated with a
milestone, for example for 2.21.2:
https://gitlab.com/lilypond/lilypond/-/milestones/320#tab-labels

Thanks,
Jonas

1: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00284.html
2: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00379.html

Hi Jonas,
I just took a quick glance on it and have many questions... :)
Let me summarize them with an example:

Take
https://gitlab.com/lilypond/lilypond/-/issues/5999

1. Federico wrote in the thread you mentioned [1]:
"In the last comment you should find a commit id (if it's missing you'll
have to search it)."

I think this refers to the time before Gitlab, when issues had
a message like

Trim unused toplevel targets.
author    Han-Wen Nienhuys 
    Sun, 22 Mar 2020 00:16:36 +0100 (00:16 +0100)
committer    Han-Wen Nienhuys 
    Tue, 31 Mar 2020 21:18:36 +0100 (22:18 +0200)
commit    0755e2f3dbb99ac256fb447d6ed14d3fe87b8ab5

at the end of the thread, right?

2. So I thought I had to look at the corresponding MRs to find
the commits I shall verify.
For #5999, there are two MRs listed as 'Related', which are:

https://gitlab.com/lilypond/lilypond/-/merge_requests/145
Add PDF bookmark support & multi-level TOC outline ability

and

https://gitlab.com/lilypond/lilypond/-/merge_requests/147
Avoid scm_append_x for property values

I'm not sure how !145 is related to #5999. Maybe #5999 did show up
after !145, and !147 did fix it.

Anyway, am I right that I have to extract all commit ids from both MRs
and check if they got into 2.21.2? Or do I have some faulty thinking here?
This would be very complicated and time-consuming, especially when MRs
consist of several commits.

3. What I do not get: A milestone is simply a named period of time, to which
you can assign MRs and issues, right? So after each release, you (or
someone else)
start a new milestone with the next version number. If an issue has been
closed, you
set the issue's milestone to the current milestone.
An issue is closed automatically when the corresponding MR has "Closes
#..." in the description.
So if an issue is closed and the milestone of the next release is added,
what is there left
to verify? Can't we be sure that the commits of the corresponding,
closed MR have gone in?

Michael

issue verification

[Jonas Hahnfeld]

Hi all,

unless I'm missing something, there has been no activity on this since
the community effort in May [1][2] and issues since version 2.21.2
don't yet have Status::Verified labels. As I'm currently updating CG to
mention our use of milestones (not sure why I forgot about this
previously), the verification procedure should probably also get
updated.
I haven't been active in this area (and don't plan to be), so it would
be great if somebody could look over the current description (assuming
the procedure should be continued).

To give a head start, here's again the list of all Status::Fixed:
https://gitlab.com/lilypond/lilypond/-/issues?scope=all=%E2%9C%93=closed_name[]=Status%3A%3AFixed

Furthermore you can directly get to the issues associated with a
milestone, for example for 2.21.2:
https://gitlab.com/lilypond/lilypond/-/milestones/320#tab-labels

Thanks,
Jonas

1: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00284.html
2: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00379.html


signature.asc
Description: This is a digitally signed message part