Re: [RFC] Enabling GitLab CI
Am Donnerstag, den 21.05.2020, 13:02 +0200 schrieb Han-Wen Nienhuys: > On Thu, May 21, 2020 at 12:34 PM Jonas Hahnfeld wrote: > > Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble: > > > On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: > > > > before merging. As we only allow fast-forward merges, this means each > > > > MR has received testing in the form it hits master. This would > > > > effectively replace the current setup of pushing to staging. > > > > > > I'm for this. > > > > Thanks. What do others (David, Han-Wen, Valentin) think about this? > > There's certainly room for improvement, but with an initial setup I can > > start writing documentation. > > Let's try it. I'll do this once all merge requests currently in Patch::push are safely in master, either later today or tomorrow. I'll try to update existing merge requests afterwards and / or give instructions. Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
Am Donnerstag, den 21.05.2020, 18:25 +0200 schrieb David Kastrup: > > If we think contention will be a problem, we cannot do the proposal. > > There's no sane "mixed bag": As outlined initially, we would 1) > > require CI for merge requests, and 2) disable direct pushes to > > master. This includes patchy which has no special permissions as far > > as GitLab is concerned. > > Sure, it would be the merge request of staging to master that would > trigger the CI then. Interesting approach, I still had patchy in the equation. We'd still need to target master initially so that James gets the CI results for the countdown process. But this could be the easiest workaround in case a developer with many patches hits contention and would be unable to merge otherwise. I'm for keeping this in mind, but not making it a rule. signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
Jonas Hahnfeld writes: > Am Donnerstag, den 21.05.2020, 17:21 +0200 schrieb Han-Wen Nienhuys: >> On Thu, May 21, 2020 at 1:17 PM James wrote: >> > On 21/05/2020 12:02, Han-Wen Nienhuys wrote: >> > > so a next step might be making the countdown process more >> > > continuous. >> > >> > What does that mean - even conceptually? >> >> My idea is that patches could enter 'countdown' stage throughout the >> day, and when they do a 48 hour period waiting period kicks in. This >> means the moment they are mergable will be more spread out throughout >> the day. >> >> My main worry is that the CI process (make doc) will be forced to run >> on machines with less CPU power, and the whole test/doc cycle would >> take 1 or even 2 hours. If CI takes 2 hours, we'd realistically only >> be able to merge 8 changes on a day. > > The full pipeline (build & test & doc) takes less than 1 hour on > GitLab's shared runners with a single core. I think it's a safe bet > that it can't get worse than that. If LilyPond itself gets slower by a > factor of 2, I'd call that a regression in general. If it is by the translation team doubling the number of supported languages... Actually, if translations were all kept up to date, we'd probably need less time for "make doc" since then most of the translations would work with the same code examples. -- David Kastrup
Re: [RFC] Enabling GitLab CI
Jonas Hahnfeld writes: > Am Donnerstag, den 21.05.2020, 17:10 +0200 schrieb David Kastrup: >> Jonas Hahnfeld writes: >> > Am Donnerstag, den 21.05.2020, 15:19 +0200 schrieb David Kastrup: >> > > Jonas Hahnfeld writes: >> > > > Am Donnerstag, den 21.05.2020, 14:29 +0200 schrieb David Kastrup: >> > > > > The "traffic jam" problem could be avoided by retaining the option of >> > > > > pushing to staging. That would occur without CI, but one could >> > > > > occasionally trigger the merge with CI on staging to have everything >> > > > > in >> > > > > it migrate to master. Since staging would be used by the more >> > > > > experienced people desiring to bunch their work before testing, the >> > > > > triggering could also happen manually by whoever thinks he has pushed >> > > > > enough stuff to staging to give it a whirl. >> > > > >> > > > That's not really how CI works. With our policy of FF merges, what >> > > > happens if some MR get merged directly to master and some sit in >> > > > staging? You'd probably rebase staging which triggers another CI >> > > > pipeline and doesn't buy you much. >> > > >> > > It buys you that several commits are bunched in staging and are treated >> > > in bulk. At least I think it does. >> > >> > No, it doesn't: The MRs must pass CI individually before it can be >> > merged. >> >> I did not propose to have CI on staging. > > In the current proposal, CI tests those merge requests that target > master. If we allowed others targeting staging without CI, we would be > unable to rely on automated testing. The automated testing would be done upon asking Gitlab to merge staging into master. > If we think contention will be a problem, we cannot do the proposal. > There's no sane "mixed bag": As outlined initially, we would 1) > require CI for merge requests, and 2) disable direct pushes to > master. This includes patchy which has no special permissions as far > as GitLab is concerned. Sure, it would be the merge request of staging to master that would trigger the CI then. > FWIW I don't see much contention at the current rate of development. Well yes. And if there were much contention, we'd not likely stay in the free plan for CI anyway. > See also my other reply to Han-Wen. -- David Kastrup
Re: [RFC] Enabling GitLab CI
Am Donnerstag, den 21.05.2020, 17:10 +0200 schrieb David Kastrup: > Jonas Hahnfeld writes: > > Am Donnerstag, den 21.05.2020, 15:19 +0200 schrieb David Kastrup: > > > Jonas Hahnfeld writes: > > > > Am Donnerstag, den 21.05.2020, 14:29 +0200 schrieb David Kastrup: > > > > > The "traffic jam" problem could be avoided by retaining the option of > > > > > pushing to staging. That would occur without CI, but one could > > > > > occasionally trigger the merge with CI on staging to have everything > > > > > in > > > > > it migrate to master. Since staging would be used by the more > > > > > experienced people desiring to bunch their work before testing, the > > > > > triggering could also happen manually by whoever thinks he has pushed > > > > > enough stuff to staging to give it a whirl. > > > > > > > > That's not really how CI works. With our policy of FF merges, what > > > > happens if some MR get merged directly to master and some sit in > > > > staging? You'd probably rebase staging which triggers another CI > > > > pipeline and doesn't buy you much. > > > > > > It buys you that several commits are bunched in staging and are treated > > > in bulk. At least I think it does. > > > > No, it doesn't: The MRs must pass CI individually before it can be > > merged. > > I did not propose to have CI on staging. In the current proposal, CI tests those merge requests that target master. If we allowed others targeting staging without CI, we would be unable to rely on automated testing. This essentially reduces the net win of the whole thing to zero. If we think contention will be a problem, we cannot do the proposal. There's no sane "mixed bag": As outlined initially, we would 1) require CI for merge requests, and 2) disable direct pushes to master. This includes patchy which has no special permissions as far as GitLab is concerned. FWIW I don't see much contention at the current rate of development. See also my other reply to Han-Wen. signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
Am Donnerstag, den 21.05.2020, 17:21 +0200 schrieb Han-Wen Nienhuys: > On Thu, May 21, 2020 at 1:17 PM James wrote: > > On 21/05/2020 12:02, Han-Wen Nienhuys wrote: > > > so a next step might be making the countdown process more > > > continuous. > > > > What does that mean - even conceptually? > > My idea is that patches could enter 'countdown' stage throughout the > day, and when they do a 48 hour period waiting period kicks in. This > means the moment they are mergable will be more spread out throughout > the day. > > My main worry is that the CI process (make doc) will be forced to run > on machines with less CPU power, and the whole test/doc cycle would > take 1 or even 2 hours. If CI takes 2 hours, we'd realistically only > be able to merge 8 changes on a day. The full pipeline (build & test & doc) takes less than 1 hour on GitLab's shared runners with a single core. I think it's a safe bet that it can't get worse than that. If LilyPond itself gets slower by a factor of 2, I'd call that a regression in general. Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
On Thu, May 21, 2020 at 1:17 PM James wrote: > > > On 21/05/2020 12:02, Han-Wen Nienhuys wrote: > > so a next step might be making the countdown process more > > continuous. > > What does that mean - even conceptually? My idea is that patches could enter 'countdown' stage throughout the day, and when they do a 48 hour period waiting period kicks in. This means the moment they are mergable will be more spread out throughout the day. My main worry is that the CI process (make doc) will be forced to run on machines with less CPU power, and the whole test/doc cycle would take 1 or even 2 hours. If CI takes 2 hours, we'd realistically only be able to merge 8 changes on a day. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Enabling GitLab CI
Jonas Hahnfeld writes: > Am Donnerstag, den 21.05.2020, 15:19 +0200 schrieb David Kastrup: >> Jonas Hahnfeld writes: >> > Am Donnerstag, den 21.05.2020, 14:29 +0200 schrieb David Kastrup: >> > > The "traffic jam" problem could be avoided by retaining the option of >> > > pushing to staging. That would occur without CI, but one could >> > > occasionally trigger the merge with CI on staging to have everything in >> > > it migrate to master. Since staging would be used by the more >> > > experienced people desiring to bunch their work before testing, the >> > > triggering could also happen manually by whoever thinks he has pushed >> > > enough stuff to staging to give it a whirl. >> > >> > That's not really how CI works. With our policy of FF merges, what >> > happens if some MR get merged directly to master and some sit in >> > staging? You'd probably rebase staging which triggers another CI >> > pipeline and doesn't buy you much. >> >> It buys you that several commits are bunched in staging and are treated >> in bulk. At least I think it does. > > No, it doesn't: The MRs must pass CI individually before it can be > merged. I did not propose to have CI on staging. > So it only buys you another test when staging progresses to master. That was the first, not another test in my plan. -- David Kastrup
Re: [RFC] Enabling GitLab CI
Am Donnerstag, den 21.05.2020, 15:19 +0200 schrieb David Kastrup: > Jonas Hahnfeld writes: > > Am Donnerstag, den 21.05.2020, 14:29 +0200 schrieb David Kastrup: > > > The "traffic jam" problem could be avoided by retaining the option of > > > pushing to staging. That would occur without CI, but one could > > > occasionally trigger the merge with CI on staging to have everything in > > > it migrate to master. Since staging would be used by the more > > > experienced people desiring to bunch their work before testing, the > > > triggering could also happen manually by whoever thinks he has pushed > > > enough stuff to staging to give it a whirl. > > > > That's not really how CI works. With our policy of FF merges, what > > happens if some MR get merged directly to master and some sit in > > staging? You'd probably rebase staging which triggers another CI > > pipeline and doesn't buy you much. > > It buys you that several commits are bunched in staging and are treated > in bulk. At least I think it does. No, it doesn't: The MRs must pass CI individually before it can be merged. So it only buys you another test when staging progresses to master. signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
Jonas Hahnfeld writes: > Am Donnerstag, den 21.05.2020, 14:29 +0200 schrieb David Kastrup: >> Jonas Hahnfeld writes: >> > Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble: >> > > On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: >> > > > before merging. As we only allow fast-forward merges, this means each >> > > > MR has received testing in the form it hits master. This would >> > > > effectively replace the current setup of pushing to staging. >> > > >> > > I'm for this. >> > >> > Thanks. What do others (David, Han-Wen, Valentin) think about this? >> > There's certainly room for improvement, but with an initial setup I can >> > start writing documentation. >> >> The "traffic jam" problem could be avoided by retaining the option of >> pushing to staging. That would occur without CI, but one could >> occasionally trigger the merge with CI on staging to have everything in >> it migrate to master. Since staging would be used by the more >> experienced people desiring to bunch their work before testing, the >> triggering could also happen manually by whoever thinks he has pushed >> enough stuff to staging to give it a whirl. > > That's not really how CI works. With our policy of FF merges, what > happens if some MR get merged directly to master and some sit in > staging? You'd probably rebase staging which triggers another CI > pipeline and doesn't buy you much. It buys you that several commits are bunched in staging and are treated in bulk. At least I think it does. > I don't mind deciding that we don't want to enable CI right now. The > purpose of bringing this up now is that I didn't want to spend time > documenting something that's going to change the next day. In my > opinion, having CI and merging to master feels more "natural" than the > staging setup. And finally if contention proves to be a problem, we > can still revert to the old setup. I was more going for the mixed bag right now :) -- David Kastrup
Re: [RFC] Enabling GitLab CI
Am Donnerstag, den 21.05.2020, 14:29 +0200 schrieb David Kastrup: > Jonas Hahnfeld writes: > > Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble: > > > On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: > > > > before merging. As we only allow fast-forward merges, this means each > > > > MR has received testing in the form it hits master. This would > > > > effectively replace the current setup of pushing to staging. > > > > > > I'm for this. > > > > Thanks. What do others (David, Han-Wen, Valentin) think about this? > > There's certainly room for improvement, but with an initial setup I can > > start writing documentation. > > The "traffic jam" problem could be avoided by retaining the option of > pushing to staging. That would occur without CI, but one could > occasionally trigger the merge with CI on staging to have everything in > it migrate to master. Since staging would be used by the more > experienced people desiring to bunch their work before testing, the > triggering could also happen manually by whoever thinks he has pushed > enough stuff to staging to give it a whirl. That's not really how CI works. With our policy of FF merges, what happens if some MR get merged directly to master and some sit in staging? You'd probably rebase staging which triggers another CI pipeline and doesn't buy you much. I generally agree that there's a potential for contention. However we're aiming for ~1h of CI which means we can merge many more patches than we currently have per countdown. Merging patches doesn't need to happen the minute James' message hits lilypond-devel, I think doing this some time until the next countdown should be fine. I don't mind deciding that we don't want to enable CI right now. The purpose of bringing this up now is that I didn't want to spend time documenting something that's going to change the next day. In my opinion, having CI and merging to master feels more "natural" than the staging setup. And finally if contention proves to be a problem, we can still revert to the old setup. Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
Jonas Hahnfeld writes: > Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble: >> On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: >> > before merging. As we only allow fast-forward merges, this means each >> > MR has received testing in the form it hits master. This would >> > effectively replace the current setup of pushing to staging. >> >> I'm for this. > > Thanks. What do others (David, Han-Wen, Valentin) think about this? > There's certainly room for improvement, but with an initial setup I can > start writing documentation. The "traffic jam" problem could be avoided by retaining the option of pushing to staging. That would occur without CI, but one could occasionally trigger the merge with CI on staging to have everything in it migrate to master. Since staging would be used by the more experienced people desiring to bunch their work before testing, the triggering could also happen manually by whoever thinks he has pushed enough stuff to staging to give it a whirl. -- David Kastrup
Re: [RFC] Enabling GitLab CI
On 21/05/2020 12:02, Han-Wen Nienhuys wrote: so a next step might be making the countdown process more continuous. What does that mean - even conceptually? The countdown is specifically to allow everyone some time to breathe and digest patches submitted without the fear of having to be always-on and missing anything. I understand the 'jam' on countdown days, but honestly how is having a countdown every day (or continuously) going to change that? I think this was touched on before, when you came back into the development group didn't seem to get the point of what my role was for in regard to countdowns. I know that other developers have been frustrated with the countdown as well (at least at first) and while I appreciate that not all patches need that much of a review, it has been evident even in these these last few weeks that some things you have felt trivial or self-evident has lead to much discussion and debate. Also note that I currently do a 2 day countdown now (mainly because I have the scope working from home), the original idea of the countdown was for 4 days as that was felt to give enough time for everyone to see a submitted patch AND give it consideration AND continue to develop at the same time. We have more devlopers and many more patches than then and we're alreadyt working on a shorter countdown. So please remember the point of countdown, it's a deliberate braking system to stop rushing things through that some might want to care about. That doesn't mean I need to do it, but neither does it mean we need 'continuous' countdown (whatever that means). Thanks. James
Re: [RFC] Enabling GitLab CI
On Thu, May 21, 2020 at 12:34 PM Jonas Hahnfeld wrote: > > Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble: > > On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: > > > before merging. As we only allow fast-forward merges, this means each > > > MR has received testing in the form it hits master. This would > > > effectively replace the current setup of pushing to staging. > > > > I'm for this. > > Thanks. What do others (David, Han-Wen, Valentin) think about this? > There's certainly room for improvement, but with an initial setup I can > start writing documentation. Let's try it. I think the FF requirement might lead to traffic jams on countdown days, so a next step might be making the countdown process more continuous. But we don't have to address that problem right now. > Jonas -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Enabling GitLab CI
Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble: > On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: > > before merging. As we only allow fast-forward merges, this means each > > MR has received testing in the form it hits master. This would > > effectively replace the current setup of pushing to staging. > > I'm for this. Thanks. What do others (David, Han-Wen, Valentin) think about this? There's certainly room for improvement, but with an initial setup I can start writing documentation. Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: > > before merging. As we only allow fast-forward merges, this means each > MR has received testing in the form it hits master. This would > effectively replace the current setup of pushing to staging. I'm for this. — Dan
Re: [RFC] Enabling GitLab CI
Am Montag, den 18.05.2020, 17:50 +0200 schrieb Jonas Hahnfeld: > Am Montag, den 18.05.2020, 12:59 +0200 schrieb Jonas Hahnfeld: > > Am Montag, den 18.05.2020, 11:53 +0100 schrieb Kevin Barry: > > > On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote: > > > > Countdown.py (which is Jonas' great cli tool) it's what you see when I > > > > do > > > > the countdown (that's literally cut/paste). > > > > > > I haven't seen that script, but the gitlab API exposes pipeline > > > information. It should be enough to correlate a merge request with a > > > pipeline status (running, pending, success, failed, etc). > > > > For reference, it's here: > > https://gitlab.com/lilypond/infrastructure/-/blob/master/countdown.py > > > > > > The API does expose this information, but the script needs to query the > > pipeline status per merge request as it's not included in the list of > > MRs. That sounds expensive, I have to experiment with this. > > See > https://gitlab.com/lilypond/infrastructure/-/merge_requests/5 > > > With a few tweaks I got this into an acceptable range, but it still > takes up to 10 seconds for ~20 merge requests. That's the cost you pay > for O(n) requests to the API and parsing all that response data... To make this less ambiguous: The time scales with the number of merge requests that have label Patch::new. Right now, this is zero (good job, James!) so for testing I waived this "optimization". signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
Am Montag, den 18.05.2020, 12:59 +0200 schrieb Jonas Hahnfeld: > Am Montag, den 18.05.2020, 11:53 +0100 schrieb Kevin Barry: > > On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote: > > > Countdown.py (which is Jonas' great cli tool) it's what you see when I do > > > the countdown (that's literally cut/paste). > > > > I haven't seen that script, but the gitlab API exposes pipeline > > information. It should be enough to correlate a merge request with a > > pipeline status (running, pending, success, failed, etc). > > For reference, it's here: > https://gitlab.com/lilypond/infrastructure/-/blob/master/countdown.py > > The API does expose this information, but the script needs to query the > pipeline status per merge request as it's not included in the list of > MRs. That sounds expensive, I have to experiment with this. See https://gitlab.com/lilypond/infrastructure/-/merge_requests/5 With a few tweaks I got this into an acceptable range, but it still takes up to 10 seconds for ~20 merge requests. That's the cost you pay for O(n) requests to the API and parsing all that response data... Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
Am Montag, den 18.05.2020, 11:53 +0100 schrieb Kevin Barry: > On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote: > > Countdown.py (which is Jonas' great cli tool) it's what you see when I do > > the countdown (that's literally cut/paste). > > I haven't seen that script, but the gitlab API exposes pipeline > information. It should be enough to correlate a merge request with a > pipeline status (running, pending, success, failed, etc). For reference, it's here: https://gitlab.com/lilypond/infrastructure/-/blob/master/countdown.py The API does expose this information, but the script needs to query the pipeline status per merge request as it's not included in the list of MRs. That sounds expensive, I have to experiment with this. signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote: > Countdown.py (which is Jonas' great cli tool) it's what you see when I do > the countdown (that's literally cut/paste). I haven't seen that script, but the gitlab API exposes pipeline information. It should be enough to correlate a merge request with a pipeline status (running, pending, success, failed, etc).
Re: [RFC] Enabling GitLab CI
On 18/05/2020 11:21, Kevin Barry wrote: On Mon, May 18, 2020 at 12:17:53PM +0200, Urs Liska wrote: No, at least not at the time I looked. What James needs is additionally an icon that states that MR is *currently* being tested. There is an icon for that (it's blue and looks like a half-filled pie chart) - I just couldn't find a merge request list that actually had any in progress. OK I don't want to go hunting if I can avoid it, and not clicking 3 or 4 pages to get to that. I have two basic methods at the moment. Countdown.py (which is Jonas' great cli tool) it's what you see when I do the countdown (that's literally cut/paste). Or *if* i really have to go into gitab's UI I live here https://gitlab.com/lilypond/lilypond/-/labels and all I do is click on the 'merge request' link and this gives me the current MR with their label. Apart from that the only time I edit or view and entire MR is to add my text/label. While it may not seem such a big deal, when you have to refresh / click different pages or scroll here and there to see a 'button' or 'flashy icon' in an already-too-busy interface, it adds a lot of friction. So the less I have to click/do before I begin the testing/countdown process the better. James
Re: [RFC] Enabling GitLab CI
On Mon, May 18, 2020 at 12:17:53PM +0200, Urs Liska wrote: > No, at least not at the time I looked. > What James needs is additionally an icon that states that MR is > *currently* being tested. There is an icon for that (it's blue and looks like a half-filled pie chart) - I just couldn't find a merge request list that actually had any in progress.
Re: [RFC] Enabling GitLab CI
Am Montag, den 18.05.2020, 11:15 +0100 schrieb Kevin Barry: > On Mon, May 18, 2020 at 10:48:55AM +0100, James Lowe wrote: > > but how do I know? That is the nub of what I am asking. If a patch > > is 'new' > > how do I know that an automated make doc is 'in progress, has > > completed with > > errors, has completed without errors' as I am not going to bother > > to do any > > work on it if it is the first two cases. What is the point? > > The merge request list has an icon next to every merge request > showing > the current pipline status. There are different icons for passed, > failed and in-progress. Is that what you want? See the merge request > page for inkscape: > https://gitlab.com/groups/inkscape/-/merge_requests > > Is the information you would like to be able to see visible there? No, at least not at the time I looked. What James needs is additionally an icon that states that MR is *currently* being tested. But that will also be visible in the interface (which Jonas addressed with his comment that in this case you can simply skip the MR and have a look later when it can be assumed finished. HTH Urs > > It will only show a pass if all stages pass, as far as I know. > > Kevin >
Re: [RFC] Enabling GitLab CI
On Mon, May 18, 2020 at 10:48:55AM +0100, James Lowe wrote: > but how do I know? That is the nub of what I am asking. If a patch is 'new' > how do I know that an automated make doc is 'in progress, has completed with > errors, has completed without errors' as I am not going to bother to do any > work on it if it is the first two cases. What is the point? The merge request list has an icon next to every merge request showing the current pipline status. There are different icons for passed, failed and in-progress. Is that what you want? See the merge request page for inkscape: https://gitlab.com/groups/inkscape/-/merge_requests Is the information you would like to be able to see visible there? It will only show a pass if all stages pass, as far as I know. Kevin
Re: [RFC] Enabling GitLab CI
On 18/05/2020 10:29, Jonas Hahnfeld wrote: Am Sonntag, den 17.05.2020, 21:14 +0100 schrieb James Lowe: On 17/05/2020 20:27, Jonas Hahnfeld wrote: - Comparison of regression tests is not yet integrated, the main problem being the need for a baseline. I already have an idea or two how this could work, but for now I'm focusing on the initial setup. This means James still needs to download the patches and run 'make check' (make doc being run automatically). Yes that would save me time. Something that strikes me immediately. * If a CI run makes doc are we going to have an entry and or label in the comment for a success/failure? I don't want to bother with any test if make/make doc fails, so I assume that it will change the label to take it off patch new. But at the same time if a patch is 'new' and hasn't had the make/make doc test, do I still run the make check anyway? If so then it passed the label would go to review and I assume the CI run would still catch that and do the make/make doc. Otherwise I am going to not only have to check for a patch 'new' I am going to have to know that it is waiting on a make doc first if you see what I mean? (will I have to second guess the CI so to speak) I'm not sure I fully understand the question(s). First of all, the system tests every new commit added to a merge request, no matter how small or if the diff is the same as before. Yes but make doc isn't instantaneous it takes time .. a good amount of time. No matter how small the diff is. (That's the advantage of computers, they don't get bored at doing the same over and over again.) If the tests pass, you get nice green ticks everywhere. On error GitLab shows red crosses instead, in particular the stage that failed (build, test, doc). This currently doesn't add the Patch::needs_work label, but I can certainly make our bot do this. Yes please, I need a simple way to know not even to bother testing a make check because if make doc fails the entire patch is useless. In some cases stuff that breaks make doc also would break 'make' (when generating our 'info' pages, some doc edits if not done correctly manifest themselves for just make rather than breaking only 'make doc). As the pipelines start immediately after the push, they might already be finished when you look at the merge request. If something didn't pass, I think it's perfectly ok to skip manual tests. but how do I know? That is the nub of what I am asking. If a patch is 'new' how do I know that an automated make doc is 'in progress, has completed with errors, has completed without errors' as I am not going to bother to do any work on it if it is the first two cases. What is the point? In case the pipelines are still running or even pending, I think you should ignore that merge request for the time being and maybe come back later. We'll see how much of an issue this turns out to be. It will be an issue. Unless it is easy for me to check. Imagine you sit down and see 20 patch-new merge requests to test. Which one do you do first? Which one has already had its make doc run? Which one has failed/passed/is in progress of its make doc run? I hope that is clearer. James
Re: [RFC] Enabling GitLab CI
Am Sonntag, den 17.05.2020, 21:14 +0100 schrieb James Lowe: > On 17/05/2020 20:27, Jonas Hahnfeld wrote: > > - Comparison of regression tests is not yet integrated, the main > > problem being the need for a baseline. I already have an idea or two > > how this could work, but for now I'm focusing on the initial setup. > > This means James still needs to download the patches and run 'make > > check' (make doc being run automatically). > > Yes that would save me time. > > Something that strikes me immediately. > > * If a CI run makes doc are we going to have an entry and or label in > the comment for a success/failure? > > I don't want to bother with any test if make/make doc fails, so I assume > that it will change the label to take it off patch new. But at the same > time if a patch is 'new' and hasn't had the make/make doc test, do I > still run the make check anyway? If so then it passed the label would go > to review and I assume the CI run would still catch that and do the > make/make doc. Otherwise I am going to not only have to check for a > patch 'new' I am going to have to know that it is waiting on a make doc > first if you see what I mean? (will I have to second guess the CI so to > speak) I'm not sure I fully understand the question(s). First of all, the system tests every new commit added to a merge request, no matter how small or if the diff is the same as before. (That's the advantage of computers, they don't get bored at doing the same over and over again.) If the tests pass, you get nice green ticks everywhere. On error GitLab shows red crosses instead, in particular the stage that failed (build, test, doc). This currently doesn't add the Patch::needs_work label, but I can certainly make our bot do this. As the pipelines start immediately after the push, they might already be finished when you look at the merge request. If something didn't pass, I think it's perfectly ok to skip manual tests. In case the pipelines are still running or even pending, I think you should ignore that merge request for the time being and maybe come back later. We'll see how much of an issue this turns out to be. Regards Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
On Sun, May 17, 2020, 3:11 PM Dan Eble wrote: > I might be willing to plug in a cheap, headless computer to crank through > patches night and day, but probably not if it will upload GBs of results. > I can also offer this, from a computer shop with lots of spare hardware and an unmetered 30 mbps up & down Internet connection. I've done various distributed computing things like Folding@Home and Storj, and this doesn't sound much harder. -- Karlin High Missouri, USA >
Re: [RFC] Enabling GitLab CI
Am Sonntag, den 17.05.2020, 16:10 -0400 schrieb Dan Eble: > On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: > > if we want to get faster builds, we can always add our own machines. > > That is a matter of installing Docker and the runner (packages provided > > by GitLab). Configuration is as simple as running one command and > > pasting the URL as well as a token. > > How much routine network traffic will the runner generate? Obviously, it > will need to pull the Docker image when it is updated, and that will happen > occasionally, but I'm more curious about the building & testing itself. Will > there be large logs or build artifacts sent back to GitLab or otherwise > served to people browsing the project? > > I might be willing to plug in a cheap, headless computer to crank through > patches night and day, but probably not if it will upload GBs of results. Obviously the builder needs to git-fetch the repo. Additionally the current configuration uploads all untracked files after the build step for usage in both test and doc. By disabling debug information, this is only 22 MB in size with at most 1x upload plus 2x download (not sure if it would cache locally). If this is of concern, we can of course fuse the three stages without artifacts in between. All log files (pure text) are uploaded to gitlab.com and served from there. After a build is finished, the builder serves no state. Also note that the runner contacts the server, ie no trouble with firewalls / NAT / dynamic IPs et al. Regards Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Enabling GitLab CI
On 17/05/2020 20:27, Jonas Hahnfeld wrote: - Comparison of regression tests is not yet integrated, the main problem being the need for a baseline. I already have an idea or two how this could work, but for now I'm focusing on the initial setup. This means James still needs to download the patches and run 'make check' (make doc being run automatically). Yes that would save me time. Something that strikes me immediately. * If a CI run makes doc are we going to have an entry and or label in the comment for a success/failure? I don't want to bother with any test if make/make doc fails, so I assume that it will change the label to take it off patch new. But at the same time if a patch is 'new' and hasn't had the make/make doc test, do I still run the make check anyway? If so then it passed the label would go to review and I assume the CI run would still catch that and do the make/make doc. Otherwise I am going to not only have to check for a patch 'new' I am going to have to know that it is waiting on a make doc first if you see what I mean? (will I have to second guess the CI so to speak) Thanks. James
Re: [RFC] Enabling GitLab CI
On May 17, 2020, at 15:27, Jonas Hahnfeld wrote: > if we want to get faster builds, we can always add our own machines. > That is a matter of installing Docker and the runner (packages provided > by GitLab). Configuration is as simple as running one command and > pasting the URL as well as a token. How much routine network traffic will the runner generate? Obviously, it will need to pull the Docker image when it is updated, and that will happen occasionally, but I'm more curious about the building & testing itself. Will there be large logs or build artifacts sent back to GitLab or otherwise served to people browsing the project? I might be willing to plug in a cheap, headless computer to crank through patches night and day, but probably not if it will upload GBs of results. — Dan
[RFC] Enabling GitLab CI
I'd like to propose that we enable GitLab CI and use it for automatically testing our merge requests. It would run 'make', 'make test', and 'make doc' like the current manual testing process. Once we have that, GitLab can be configured to enforce a successful pipeline before merging. As we only allow fast-forward merges, this means each MR has received testing in the form it hits master. This would effectively replace the current setup of pushing to staging. The system would use Docker containers which is similar in spirit to Han-Wen's proposal in February [1]. However the system is much simpler and the containers don't come with a full development environment. Instead they have barely enough to compile and test LilyPond. For the actual changes and to get a feel of the UX, see the MR: https://gitlab.com/lilypond/lilypond/-/merge_requests/57 Advantages == - We get automatic testing, tightly integrated into GitLab. - It helps the Patch Meister (James); see below for one caveat. - No patchy required for pushing to master. Disadvantages = - Using Docker images, we get a single standardized environment for the tests. While being great for reproducibility, it looses the diversity we had so far (please refer to earlier discussions). - Comparison of regression tests is not yet integrated, the main problem being the need for a baseline. I already have an idea or two how this could work, but for now I'm focusing on the initial setup. This means James still needs to download the patches and run 'make check' (make doc being run automatically). - There's no possibility to push without a merge request, and it needs to be merged through the UI. I don't consider this a loss personally. - There's no possibility to push multiple changes at once: Merge requests are rebased, tested and merged one by one. GitLab is planning to implement a queuing mechanism, and if all else fails we can invent our own scripts. Where to Run Now towards infrastructure: Being a public project and open-source GitLab currently offers 50,000 CI minutes per month for free. [1] This has the advantage that test results are available immediately. 50,000 minutes divided by ~60 minutes per build, this is ~800 builds per month, or around 25 per day. Not sure what happens if we exceed this, comments claim it's "enforced" in some way. However there's no page to see the current usage (for a public group project)... If this is too little, if GitLab reduces the amount of free minutes, or if we want to get faster builds, we can always add our own machines. That is a matter of installing Docker and the runner (packages provided by GitLab). Configuration is as simple as running one command and pasting the URL as well as a token. We can even go hybrid and use the best of both worlds. My tests suggest that specific runners are preferred, using the shared runners as fallback. Let me know what you think Jonas 1: https://lists.gnu.org/archive/html/lilypond-devel/2020-02/msg00416.html 2: Our 'lilypond' group is older than https://about.gitlab.com/releases/2020/03/18/ci-minutes-for-free-users/ signature.asc Description: This is a digitally signed message part
