Re: CI buildbots

[Steven Schveighoffer via Digitalmars-d]

On 5/23/18 4:51 PM, Manu wrote:

I'm not suggesting adding new machines... I'm suggesting removing the
ones that take ~50 minutes.


I think this thought has merit. Or at least delegate them to only test 
older PRs.



I guess the flow I observed on the weekend, where it took me 3 days to
get my batch of PR's merged, is that people reviewed it once it was
already green... so that's at least 50 minutes at the end of my actual
work, THEN they ask me to add `package` or `const` somewhere, so I do,


I don't think everyone does this, and I'd be surprised at any who do, 
with the caveat that if a PR is for a specific platform, they may wait 
to ensure the PR passes that platform before approving.


My workflow is to FIRST look at the PR and see if it looks good, then if 
needed, take a look at the auto tester. Hell, I often times approve a PR 
with an obvious syntax error that I didn't notice, only to see the auto 
tester fail later.


So it was probably a coincidence that they asked for the changes after 
the tests were complete.



and that's another hour before they have another look to confirm the
tweak, and then they probably got bored of reviewing PR's in the last
2-3 hours, and went to bed, or to work, or to get lunch or something,
so they reappear the next day.
So in practise, adding 2-3 hours to the cycles adds 24 hours to the
cycle. The latency was the problem for all of my 5-6 patches, not the
throughput.


I'm going to level with you, not everyone is on Manu's schedule to get 
things done this weekend. Sorry. This is open-source/volunteer workflow. 
I would be more likely to believe it's just people are going on their 
own schedule when they can donate their time. I submitted a PR at the 
hackathon, and it was approved as I was flying home. Still not merged 
(2+ weeks later). I don't think it has anything to do with the tester.


If these were DMD PRs, you are in for latency that has nothing to do 
with test machines -- there just aren't a lot of people who are 
qualified to or feel comfortable with reviewing PRs.


And ping people who have reviewed, or who you think might be inclined to 
merge. Sqeaky wheels and all that.



Maybe those 4 machines could be assigned a rule, where they're only
assigned jobs to re-test PR's with updated DMD's or whatever if the PR
has been silent for >24 hours or something... so they are assigned to
low-frequency jobs, and never assigned to fresh jobs? 


Yes, I think this idea makes sense. It's really up to Brad though.

-Steve

Re: CI buildbots

[Manu via Digitalmars-d]

On 23 May 2018 at 07:09, Seb via Digitalmars-d
 wrote:
> On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:
>>
>> This CI situation with the DMD/druntime repos is not okay.
>> It takes ages... **hours** sometimes, for CI to complete.
>> It's all this 'auto-tester' one, which seems to lock up on the last few
>> tests.
>>
>> This makes DMD is a rather unenjoyable project to contribute to.
>> I had a sudden burst of inspiration, but it's very rapidly wearing off.
>
>
> As mentioned on GitHub, running the compile+fail testsuite locally takes 10
> seconds.
> Typically a PR doesn't touch much cross-platform stuff and if it does, you
> should get a negative reply pretty early from any CI.
> If you use CIs to detect merge conflicts, they will also occur locally.
>
> As explained on GitHub auto-tester gives priority to PRs with the auto-merge
> label and will constantly invalidate old builds whenever something got
> merged, so it typically never completes for a PR.
> OTOH if your PR has been approved, it will have priority access and normally
> it will be merged quite quickly.
>
> That being said, if you have ideas on how to improve the ecosystem, please
> let us/me know (except for adding new machines to the auto-tester - that's
> something that seems to be out of our reach).

I'm not suggesting adding new machines... I'm suggesting removing the
ones that take ~50 minutes.
I think they're a let loss for the pipeline. A <10 minute machine
could finish up 4 other jobs AND finish mine in the same amount of
time. It practise it's likely to get to mine a lot sooner.
A single machine in the pipeline that takes 50 minutes makes the
pipeline take at least 50 minutes.

Latency > throughput, the pipeline would be better without those 4
machines. (win-farm-1, win-farm-2, bellevue, inglebrook)

I guess the flow I observed on the weekend, where it took me 3 days to
get my batch of PR's merged, is that people reviewed it once it was
already green... so that's at least 50 minutes at the end of my actual
work, THEN they ask me to add `package` or `const` somewhere, so I do,
and that's another hour before they have another look to confirm the
tweak, and then they probably got bored of reviewing PR's in the last
2-3 hours, and went to bed, or to work, or to get lunch or something,
so they reappear the next day.
So in practise, adding 2-3 hours to the cycles adds 24 hours to the
cycle. The latency was the problem for all of my 5-6 patches, not the
throughput.

Maybe those 4 machines could be assigned a rule, where they're only
assigned jobs to re-test PR's with updated DMD's or whatever if the PR
has been silent for >24 hours or something... so they are assigned to
low-frequency jobs, and never assigned to fresh jobs?

Re: CI buildbots

[Seb via Digitalmars-d]

On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:

This CI situation with the DMD/druntime repos is not okay.
It takes ages... **hours** sometimes, for CI to complete.
It's all this 'auto-tester' one, which seems to lock up on the 
last few tests.


This makes DMD is a rather unenjoyable project to contribute to.
I had a sudden burst of inspiration, but it's very rapidly 
wearing off.


As mentioned on GitHub, running the compile+fail testsuite 
locally takes 10 seconds.
Typically a PR doesn't touch much cross-platform stuff and if it 
does, you should get a negative reply pretty early from any CI.
If you use CIs to detect merge conflicts, they will also occur 
locally.


As explained on GitHub auto-tester gives priority to PRs with the 
auto-merge label and will constantly invalidate old builds 
whenever something got merged, so it typically never completes 
for a PR.
OTOH if your PR has been approved, it will have priority access 
and normally it will be merged quite quickly.


That being said, if you have ideas on how to improve the 
ecosystem, please let us/me know (except for adding new machines 
to the auto-tester - that's something that seems to be out of our 
reach).

Re: CI buildbots

[Jonathan Marler via Digitalmars-d]

On Monday, 21 May 2018 at 22:21:34 UTC, Manu wrote:
On 21 May 2018 at 09:22, Jonathan Marler via Digitalmars-d 
 wrote:

On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:


This CI situation with the DMD/druntime repos is not okay.
It takes ages... **hours** sometimes, for CI to complete.
It's all this 'auto-tester' one, which seems to lock up on 
the last few

tests.

This makes DMD is a rather unenjoyable project to contribute 
to.
I had a sudden burst of inspiration, but it's very rapidly 
wearing off.



It might take hours for CI to complete, but it can take weeks 
or months for someone to review your code...so the CI time 
doesn't really seem to matter for myself.  That is unless 
you're trying to use the CI in your modify/test development 
cycle.  However, that's should be solvable by testing locally 
in most cases.


I use CI to test the platforms I don't build locally. That's 
natural for cross-platform development.


Ah I see.  Well for me it seemed worth the effort to setup at 
least a windows and linux machine which covers most testing.  The 
worst was when we were having intermittent seg faults on 32-bit 
OSx...I eventually borrowed my girlfriends macbook to reproduce 
and debug that one...ugh that was a pain.  In any case I'd 
recommend having one posix and one windows platform.  There's 
always VMs if you need em :)


But to address your original concern, I'm not sure that 
decreasing the testing required to integrate changes is 
necessarily a net positive.  If you can decrease test time while 
maintaining the same coverage then by all means, let's do that!  
But I think in general you have to find a balance between the two.


Since you are using CI for your modify/build/test development 
cycle, one idea would be to define some sort of interface that 
the CI's could use to limit what they are testing for a 
particular PR.  You could have it search through the comments for 
something like "limit test to dmd runnable" or something like 
that.

Re: CI buildbots

[Manu via Digitalmars-d]

On 21 May 2018 at 09:22, Jonathan Marler via Digitalmars-d
 wrote:
> On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:
>>
>> This CI situation with the DMD/druntime repos is not okay.
>> It takes ages... **hours** sometimes, for CI to complete.
>> It's all this 'auto-tester' one, which seems to lock up on the last few
>> tests.
>>
>> This makes DMD is a rather unenjoyable project to contribute to.
>> I had a sudden burst of inspiration, but it's very rapidly wearing off.
>
>
> It might take hours for CI to complete, but it can take weeks or months for
> someone to review your code...so the CI time doesn't really seem to matter
> for myself.  That is unless you're trying to use the CI in your modify/test
> development cycle.  However, that's should be solvable by testing locally in
> most cases.

I use CI to test the platforms I don't build locally. That's natural
for cross-platform development.

Re: CI buildbots

[Jonathan Marler via Digitalmars-d]

On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:

This CI situation with the DMD/druntime repos is not okay.
It takes ages... **hours** sometimes, for CI to complete.
It's all this 'auto-tester' one, which seems to lock up on the 
last few tests.


This makes DMD is a rather unenjoyable project to contribute to.
I had a sudden burst of inspiration, but it's very rapidly 
wearing off.


It might take hours for CI to complete, but it can take weeks or 
months for someone to review your code...so the CI time doesn't 
really seem to matter for myself.  That is unless you're trying 
to use the CI in your modify/test development cycle.  However, 
that's should be solvable by testing locally in most cases.

Re: CI buildbots

[Steven Schveighoffer via Digitalmars-d]

On 5/21/18 3:21 AM, Manu wrote:

On 20 May 2018 at 21:46, Manu  wrote:

This CI situation with the DMD/druntime repos is not okay.
It takes ages... **hours** sometimes, for CI to complete.
It's all this 'auto-tester' one, which seems to lock up on the last few tests.

This makes DMD is a rather unenjoyable project to contribute to.
I had a sudden burst of inspiration, but it's very rapidly wearing off.


A few of those machines can build AND run the tests in 5-6 mintues. A
lot under 10 minutes...
Then there's a few that take 45+ minutes. Why are they in the pool? Is
it really worth having 20 machines build the thing, especially when a
few of them blow the turn-around time to into hours rather than
minutes?


IMO -- yes. These are machines that are donated to dlang foundation 
essentially. The more machines that are in the fleet, the more tests can 
be run. One of these is actually a VM running on my desktop at home 
(which pretty much sits idle all day otherwise). And it's not in the 45+ 
minute category, but is definitely not in the 10 minute category either. 
There are also different configurations on some of these machines that 
it's good to test with.


That being said, I wonder if it wouldn't be good to prioritize less 
stale PRs to the fastest machines. I'm not sure exactly how the auto 
tester assigns PRs to test machines.


-Steve

Re: CI buildbots

[Mike Franklin via Digitalmars-d]

On Monday, 21 May 2018 at 07:21:31 UTC, Manu wrote:

A few of those machines can build AND run the tests in 5-6 
mintues. A

lot under 10 minutes...
Then there's a few that take 45+ minutes. Why are they in the 
pool? Is
it really worth having 20 machines build the thing, especially 
when a

few of them blow the turn-around time to into hours rather than
minutes?


I'm not sure of the best way to elicit action on this issue, but 
if it were me, I'd try filing and issue at 
https://github.com/braddr/d-tester/issues and see what Brad says.


Mike

Re: CI buildbots

[Manu via Digitalmars-d]

On 20 May 2018 at 21:46, Manu  wrote:
> This CI situation with the DMD/druntime repos is not okay.
> It takes ages... **hours** sometimes, for CI to complete.
> It's all this 'auto-tester' one, which seems to lock up on the last few tests.
>
> This makes DMD is a rather unenjoyable project to contribute to.
> I had a sudden burst of inspiration, but it's very rapidly wearing off.

A few of those machines can build AND run the tests in 5-6 mintues. A
lot under 10 minutes...
Then there's a few that take 45+ minutes. Why are they in the pool? Is
it really worth having 20 machines build the thing, especially when a
few of them blow the turn-around time to into hours rather than
minutes?

CI buildbots

[Manu via Digitalmars-d]

This CI situation with the DMD/druntime repos is not okay.
It takes ages... **hours** sometimes, for CI to complete.
It's all this 'auto-tester' one, which seems to lock up on the last few tests.

This makes DMD is a rather unenjoyable project to contribute to.
I had a sudden burst of inspiration, but it's very rapidly wearing off.