Re: Autotesting dub packages with dmd nightly

2016-08-27 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Friday, 26 August 2016 at 18:52:17 UTC, Seb wrote:

That's awesome to know!

How difficult would it be to integrate it with the dlang GitHub 
PR workflow?


Not at all. Just need an api key from someone with administration 
access. But let's not get ahead of ourselves. Right now I am just 
planning to contact the github api.


Still, there is some work to be done first for purging and 
updating the job queue when prs are updated and probably some 
other cases.


Also there are some choices left regarding the interpretation of 
the results. Right now, for pull requests I do a diff with the 
latest dmd release and collect all the packages that have a 
different outcome. It would be better to run the comparison 
against the commit the pull request was based on, although that 
requires building twice as much.


Currently I am focused on a simple frontend to give people a view 
into the results. It is coming along quite nicely.



I am just shooting an idea that popped into my head:

We already use CircleCi and Travis for the dlang repos, so if 
we lock the packages to a fixed version (to prevent failures 
caused by the package authors), we might be able create a 
simple file like:


```
vibe.d==0.7.29
mir==0.16.3
...
```

We could select a subset (e.g. 50-100), s.t. the runtime 
doesn't get exorbitant.
Following we could then do enable the checking in CircleCi with 
sth. similar to:


```
wget 
https://raw.githubusercontent.com/dlang/community-list/master/dlang-packages.master # just an example, idea is to host the file somewhere on GitHub s.t. editing it is easy

dub fetch your-fancy-tool --version="x.y.y"
dub run your-fancy-tool --config dlang-stable.packages
```

Of course CircleCi doesn't have the access rights to post back 
to the hook API, but you could send a notification to dlang-bot 
[1] which has the permissions or let the CI error / fail.


Otherwise you could of course look into setting up your own job 
queue (or hack with the code from the auto-tester [2]), which 
might be fun.
I already have my own queue. The important part though is a place 
to keep the results and running queries against it. Currently I 
use various regexes against dub test's output and categorise 
accordingly.


What I am really happy about is the aggressive caching. It allows 
me to build and unit tests a package in 10sec on average.


Re: Autotesting dub packages with dmd nightly

2016-08-26 Thread Seb via Digitalmars-d-announce

On Monday, 22 August 2016 at 20:44:05 UTC, Sebastiaan Koppe wrote:
On Wednesday, 10 August 2016 at 18:35:03 UTC, Sebastiaan Koppe 
wrote:

So true. Then I will do PR's first.


I finally got around implementing running dmd/druntime/phobos 
pull requests against all dub packages. Thank you digger, for 
making it so easy.


I did a run of master + druntime#1602. Then I ran a comparison 
with a batch from 2.071.2-b2. 108 packages had a different 
build result. I have no nice stats or pictures, but a quick 
glance over the raw data:


50 of them went from green unittests to a dmd exit code 1.
16 went from unknown build results to dmd exit code 1.
10 went from dmd exit code 255 to 1
9 of them are now green.
8 of them went from linker errors to a dmd exit code 1.
6 of then went from non-zero exit code during unittest run to a 
dmd exit code 1.
3 previous ran out of memory but now resulted in dmd exit code 
1.

etc.

All in all I think +/- 96 package are affected. A little over 
11%.


That's awesome to know!

How difficult would it be to integrate it with the dlang GitHub 
PR workflow?

I am just shooting an idea that popped into my head:

We already use CircleCi and Travis for the dlang repos, so if we 
lock the packages to a fixed version (to prevent failures caused 
by the package authors), we might be able create a simple file 
like:


```
vibe.d==0.7.29
mir==0.16.3
...
```

We could select a subset (e.g. 50-100), s.t. the runtime doesn't 
get exorbitant.
Following we could then do enable the checking in CircleCi with 
sth. similar to:


```
wget 
https://raw.githubusercontent.com/dlang/community-list/master/dlang-packages.master # just an example, idea is to host the file somewhere on GitHub s.t. editing it is easy

dub fetch your-fancy-tool --version="x.y.y"
dub run your-fancy-tool --config dlang-stable.packages
```

Of course CircleCi doesn't have the access rights to post back to 
the hook API, but you could send a notification to dlang-bot [1] 
which has the permissions or let the CI error / fail.


Otherwise you could of course look into setting up your own job 
queue (or hack with the code from the auto-tester [2]), which 
might be fun too.


[1] https://github.com/MartinNowak/dlang-bot
[2] https://github.com/braddr/d-tester


Re: Autotesting dub packages with dmd nightly

2016-08-22 Thread Sebastiaan Koppe via Digitalmars-d-announce
On Wednesday, 10 August 2016 at 18:35:03 UTC, Sebastiaan Koppe 
wrote:

So true. Then I will do PR's first.


I finally got around implementing running dmd/druntime/phobos 
pull requests against all dub packages. Thank you digger, for 
making it so easy.


I did a run of master + druntime#1602. Then I ran a comparison 
with a batch from 2.071.2-b2. 108 packages had a different build 
result. I have no nice stats or pictures, but a quick glance over 
the raw data:


50 of them went from green unittests to a dmd exit code 1.
16 went from unknown build results to dmd exit code 1.
10 went from dmd exit code 255 to 1
9 of them are now green.
8 of them went from linker errors to a dmd exit code 1.
6 of then went from non-zero exit code during unittest run to a 
dmd exit code 1.

3 previous ran out of memory but now resulted in dmd exit code 1.
etc.

All in all I think +/- 96 package are affected. A little over 11%.


Re: Autotesting dub packages with dmd nightly

2016-08-10 Thread Seb via Digitalmars-d-announce
On Wednesday, 10 August 2016 at 18:35:03 UTC, Sebastiaan Koppe 
wrote:
Yes, but from past experience we know that people don't look 
at results, if you don't make it part of PR acceptance.


So true. Then I will do PR's first.


Thinking about it, you could also opt for integrating it with the 
dmd PR flow - in a similar manner to the autotester or coverage 
bot:
Select a subset (depending on the runtime) of packages and run 
your dub autotester for every commit and thus save for every 
commit a list of passing packages.
Now for a new PR, search for the master commit hash in your DB of 
runs and run the dub autotester with those packages. The workflow 
of the AutoTester [1] is a bit more complicated, because it is 
throwing away results as soon as the master HEAD changes (to 
avoid any inconsistencies) and there are often rebases and 
additional pushes happening, but you could just opt for a simple 
80% solution.
I imagine shouting at Walter with a Github comment "Hey this PR 
will break 10% of all packages [of the subset]" could be quite 
helpful ;-)


[1] https://auto-tester.puremagic.com


Re: Autotesting dub packages with dmd nightly

2016-08-10 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Wednesday, 10 August 2016 at 10:32:24 UTC, Martin Nowak wrote:
We want better ranking of dub packages (mostly by download, but 
for sure also showing CI results [¹]).


I was also thinking about integrating results from CI builds that 
packages do themselves. But there is some 'impedance mismatch': 
those CI build are done on the master branch, not on the latest 
release that is on code.dlang.org.



Is it already usable?


Short answer: No.

I am currently test running it on all packages against the 10 
latest dmd releases (I have done 6k packages on and off since 2 
days ago). But I am running into vibe.d issues/missing features. 
Things like not being able to use gzip with requestHttp (let 
alone with a RestInterfaceClient), invalid internal state with 
the http client pool on interrupting requests, and some other 
things.


Also, I am writing a PR for vibe.d to send http request to unix 
sockets.



How to deploy then?


For the worker it's just a docker container. But until the unix 
sockets PR is done you do have to setup the docker daemon to 
listen on the docker0 interface.



I need to test
https://github.com/dlang/druntime/pull/1602 and otherwise have 
to resetup my project tester for that.


I am using digger to build dmd, so adding in the pull request is 
trivial. I do need to adjust internals to properly handle it 
though.


But alas, family is coming over so don't expect anything anytime 
soon.


Yes, but from past experience we know that people don't look at 
results, if you don't make it part of PR acceptance.


So true. Then I will do PR's first.


Re: Autotesting dub packages with dmd nightly

2016-08-10 Thread Martin Nowak via Digitalmars-d-announce
On 08/08/2016 09:54 AM, Sebastiaan Koppe wrote:
> On Sunday, 7 August 2016 at 23:08:34 UTC, Martin Nowak wrote:
>> I actually don't think this makes sense. You're not in the position to
>> maintain 1K+ packages, it's the library owners that need to test their
>> code.
> Thanks for taking the time to respond.

You're welcome. This is an important topic for us.

> I agree with you. Library owners should test their code themselves. But
> they don't. 24% of the packages don't build.

We want better ranking of dub packages (mostly by download, but for sure
also showing CI results [¹]). It's rather trivial to filter out
low-quality packages b/c they're hardly used.

[¹]: https://trello.com/c/CaYJwtBV/60-integrate-ci-results-with-dub-registry

>> Just this short list I'm using for the project tester is hardly
>> maintainable.
> I don't need to maintain anything besides linker errors. It is quite
> simple, I just run `dub test` and see what happens. If that doesn't work
> I consider it a failed build.
> 
>> https://github.com/MartinNowak/project_tester (uses Jenkins, no need
>> to write yet another CI).
> I would argue mine is simpler to deploy and have nodes join.

Is it already usable? How to deploy then? I need to test
https://github.com/dlang/druntime/pull/1602 and otherwise have to
resetup my project tester for that.
Adding more servers to Jenkins is trivial as well.

>> Once a day is not enough b/c will feel responsible for breakages, we
>> really need feedback before merging.
> It is just a matter of resources. I choose nightly since it seemed
> doable using just my own resources.

Yes, but from past experience we know that people don't look at results,
if you don't make it part of PR acceptance.

>> - Show test results of various CIs on code.dlang.org. Testing a dub
>> package on Travis-CI is already a no-brainer. For example the
>> following .travis.yml would test a package against all dmd release
>> channels.
>>
>> ```yaml
>> language: d
>> d: [dmd, dmd-beta, dmd-nightly]
>> ```
> Yes, that is quite nice. But that only gets triggered when the repo is
> updated.

Travis now allows cron scheduling, you still have to ask their support
to unlock that.

-Martin


Re: Autotesting dub packages with dmd nightly

2016-08-08 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Sunday, 7 August 2016 at 23:08:34 UTC, Martin Nowak wrote:
I actually don't think this makes sense. You're not in the 
position to maintain 1K+ packages, it's the library owners that 
need to test their code.

Thanks for taking the time to respond.

I agree with you. Library owners should test their code 
themselves. But they don't. 24% of the packages don't build.


Just this short list I'm using for the project tester is hardly 
maintainable.
I don't need to maintain anything besides linker errors. It is 
quite simple, I just run `dub test` and see what happens. If that 
doesn't work I consider it a failed build.


https://github.com/MartinNowak/project_tester (uses Jenkins, no 
need to write yet another CI).

I would argue mine is simpler to deploy and have nodes join.

I've already thought about many different aspects of this and 
here are the 2 things that are useful and might work out.
- Implement a tester that runs for every PR (just the other 
testers) and tests the most popular/important dub packages.
Once a day is not enough b/c will feel responsible for 
breakages, we really need feedback before merging.
It is just a matter of resources. I choose nightly since it 
seemed doable using just my own resources.


- Show test results of various CIs on code.dlang.org. Testing a 
dub package on Travis-CI is already a no-brainer. For example 
the following .travis.yml would test a package against all dmd 
release channels.


```yaml
language: d
d: [dmd, dmd-beta, dmd-nightly]
```
Yes, that is quite nice. But that only gets triggered when the 
repo is updated.


All in all I understand your reservations, and I highly 
appreciate your feedback. I understand I won't bring the end-all 
solution to testing, but I do hope to reach the goals that I have 
set forth for myself. 1) catching (some) regressions, 2) giving 
insights into bit rot on code.dlang.org, 3) have fun.


It might take a couple of months before I reach them, or I might 
not at all.


Re: Autotesting dub packages with dmd nightly

2016-08-07 Thread Martin Nowak via Digitalmars-d-announce

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
I am aiming really low at first, but will eventually add things 
like memory usage, history, notifications, etc.


I actually don't think this makes sense. You're not in the 
position to maintain 1K+ packages, it's the library owners that 
need to test their code.
Just this short list I'm using for the project tester is hardly 
maintainable.
https://github.com/MartinNowak/project_tester (uses Jenkins, no 
need to write yet another CI).


I've already thought about many different aspects of this and 
here are the 2 things that are useful and might work out.


- Implement a tester that runs for every PR (just the other 
testers) and tests the most popular/important dub packages. Once 
a day is not enough b/c will feel responsible for breakages, we 
really need feedback before merging.


- Show test results of various CIs on code.dlang.org. Testing a 
dub package on Travis-CI is already a no-brainer. For example the 
following .travis.yml would test a package against all dmd 
release channels.


```yaml
language: d
d: [dmd, dmd-beta, dmd-nightly]
```


Re: Autotesting dub packages with dmd nightly

2016-08-07 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Saturday, 6 August 2016 at 20:00:53 UTC, Basile B. wrote:
On Saturday, 6 August 2016 at 19:06:34 UTC, Sebastiaan Koppe 
wrote:
- code.dlang.org has an api but doesn't provide an endpoint to 
retrieve all packages/version. Now I just scrape the site 
instead (thanks Adam for your dom implementation).


No endpoint but still possible in two steps.
Yeah, I considered something like that myself. I find scraping to 
be better for both sides though, and simpler. A good api would 
certainly be better.


On Saturday, 6 August 2016 at 20:08:47 UTC, Seb wrote:
Why don't you make a PR to the dub registry 
(https://github.com/dlang/dub-registry) to get such an 
endpoint? Or at least open an issue ;-)

https://github.com/dlang/dub-registry/issues/171


Re: Autotesting dub packages with dmd nightly

2016-08-07 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Saturday, 6 August 2016 at 19:46:52 UTC, Seb wrote:

That are excellent news!

Thanks.

1) Send the packages a notification about the build error (e.g. 
Github comment) - this should probably be tweaked a bit, s.t. 
it doesn't spam too often for still broken packages
I was thinking about having people register for notifications 
themselves.


2) Allow easy, manual build of special branches for the core 
team.
I need something similar for dev/testing purposes as well. Since 
I am using digger it is really easy to build whatever dmd + pull 
request is needed. Problem is controlling access.



3) Once you have the API
a) (try to) get a shield badge (-> http://shields.io/)

Nice find. Will use.

b) Make the data available to the dub-registry (-> 
https://github.com/dlang/dub-registry)

Sure.

4) Assess the quality of the unittests. Probably the easiest is 
`dub test -b unittest-cov`, and then summing up the total 
coverage of all generated .lst files
I am not sure this is a good idea. Besides the fact that coverage 
doesn't correlate with quality, it is outside of the purpose for 
this tool (identifying dmd regressions and identifying broken 
packages).


5) Log your daily "broken" statistics - could be a good 
indicator of whether your hard work gets acknowledged.

I rather hear it from people than seeing it in the stats :)

6) Regarding linker errors - I can only redirect you to the 
open DUB issue (https://github.com/dlang/dub/issues/852) and 
DEP 5 (https://github.com/dlang/dub/wiki/DEP5).
It is an open problem and I don't wont to solve it. For now I 
think I will just install the most important ones and just accept 
that not all packages will be build.


On another note, I do think the dub package definition could use 
some extra fields. Like compatible platforms and compatible dmd 
versions. Take vibe.d for instance, it is specifically build for 
certain dmd versions and it makes no sense for dubster to try to 
compile it with an unsupported version. Also, it allows dub 
itself to notify you of incompatible packages w.r.t. the 
installed compiler. Same idea for platform.


Re: Autotesting dub packages with dmd nightly

2016-08-06 Thread Seb via Digitalmars-d-announce
On Saturday, 6 August 2016 at 19:06:34 UTC, Sebastiaan Koppe 
wrote:
- code.dlang.org has an api but doesn't provide an endpoint to 
retrieve all packages/version. Now I just scrape the site 
instead (thanks Adam for your dom implementation).


Why don't you make a PR to the dub registry 
(https://github.com/dlang/dub-registry) to get such an endpoint? 
Or at least open an issue ;-)


Re: Autotesting dub packages with dmd nightly

2016-08-06 Thread Seb via Digitalmars-d-announce
On Saturday, 6 August 2016 at 19:06:34 UTC, Sebastiaan Koppe 
wrote:
I have just finished a first iteration of dubster, a test 
runner that runs `dub test` on each package for each dmd 
release.


see https://github.com/skoppe/dubster

Please provide feedback as it will determine the direction/life 
of this tester.


I am planning on adding a web ui/api next to look around in the 
data.


That are excellent news!
Some random ideas:

1) Send the packages a notification about the build error (e.g. 
Github comment) - this should probably be tweaked a bit, s.t. it 
doesn't spam too often for still broken packages
2) Allow easy, manual build of special branches for the core 
team, e.g. let's say Walter develops the new scoped pointers 
feature ( https://github.com/dlang/DIPs/pull/24), than it would 
be great to know how many packages would break by pulling in the 
branch (in comparison to the last release or current nightly). A 
similar "breakage by shipping" test might be very interesting for 
critical changes to druntime or phobos too.

3) Once you have the API
a) (try to) get a shield badge (-> http://shields.io/)
b) Make the data available to the dub-registry (-> 
https://github.com/dlang/dub-registry)
4) Assess the quality of the unittests. Probably the easiest is 
`dub test -b unittest-cov`, and then summing up the total 
coverage of all generated .lst files, but running with coverage 
might increase your build times, though I would argue that it's 
worth it ;-)
5) Log your daily "broken" statistics - could be a good indicator 
of whether your hard work gets acknowledged.
6) Regarding linker errors - I can only redirect you to the open 
DUB issue (https://github.com/dlang/dub/issues/852) and DEP 5 
(https://github.com/dlang/dub/wiki/DEP5).


Re: Autotesting dub packages with dmd nightly

2016-07-19 Thread vladdeSV via Digitalmars-d-announce

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
Just to let you guys know - and to be sure no one is doing the 
same - I decided to go ahead and *start* writing an autotester 
that will fetch dmd nightly and unittest each dub package.


It will be using a classic master-worker architecture and will 
leverage docker containers.


I am aiming really low at first, but will eventually add things 
like memory usage, history, notifications, etc.


Nice :)


Re: Autotesting dub packages with dmd nightly

2016-07-18 Thread Jacob Carlborg via Digitalmars-d-announce

On 2016-07-16 22:34, Sebastiaan Koppe wrote:

Just to let you guys know - and to be sure no one is doing the same - I
decided to go ahead and *start* writing an autotester that will fetch
dmd nightly and unittest each dub package.

It will be using a classic master-worker architecture and will leverage
docker containers.

I am aiming really low at first, but will eventually add things like
memory usage, history, notifications, etc.


Just as a test I setup a project on GitLab.com [1]. This is an example 
of a build [2]. The config for building that looks like:


linux:
  script:
- curl -L -o dvm 
https://github.com/jacob-carlborg/dvm/releases/download/v0.4.4/dvm-0.4.4-linux-debian7-x86_64

- chmod +x dvm
- ./dvm install dvm
- source ~/.bashrc
- dvm install 2.071.1
- dvm use 2.071.1 -d
- curl -o dub.tar.gz 
'http://code.dlang.org/files/dub-1.0.0-linux-x86_64.tar.gz'

- tar -xzf dub.tar.gz
- ./dub test

"linux" is the name of the job and "script" is the commands that should 
be executed.


The setup I was thinking about would work something like this:

1. Create one or more runners for each supported platform
2. Tag those in GitLab with the name of the platform
3. Create one job per platform that should be tested
4. Use the tags to specify which runner should be used, something like 
this [3]:


linux_x86_64:
  tags:
- linux_x86_64
  script:
- ./gitlab.sh

windows:
  tag:
- windows
  script: gitlab.bat

osx:
  tag:
- osx
  script: ./gitlab.sh

5. Update code.dlang.org to mirror the repository to GitLab when it 
finds an update in GitHub

6. Trigger a build using the GitLab API [4]:

curl -X POST \
  -F token=TOKEN \
  -F ref=master \
  https://gitlab.example.com/api/v3/projects/9/trigger/builds

One issue is to get the .gitlab-ci.yml file in the repository.

Since it's possible to host GitLab and the runners you get full control 
of everything.


[1] https://gitlab.com/Carlborg/orange
[2] https://gitlab.com/Carlborg/orange/builds/2450044
[3] https://gitlab.com/help/ci/yaml/README.md
[4] https://gitlab.com/help/ci/triggers/README.md

--
/Jacob Carlborg


Re: Autotesting dub packages with dmd nightly

2016-07-18 Thread Edwin van Leeuwen via Digitalmars-d-announce

On Monday, 18 July 2016 at 18:47:28 UTC, Jacob Carlborg wrote:

On 2016-07-18 11:55, Sebastiaan Koppe wrote:

Like I said I am aiming really low. On purpose. I have a wife 
and two

kids and I need to keep the scope limited.


In that case, go with something that already exists.


I think Martin Nowak has some sort of automated setup for testing 
a limited number of dub packages against each release, but I 
can't find the relevant post at the moment.


Re: Autotesting dub packages with dmd nightly

2016-07-18 Thread Jacob Carlborg via Digitalmars-d-announce

On 2016-07-18 11:55, Sebastiaan Koppe wrote:


Like I said I am aiming really low. On purpose. I have a wife and two
kids and I need to keep the scope limited.


In that case, go with something that already exists.

--
/Jacob Carlborg


Re: Autotesting dub packages with dmd nightly

2016-07-18 Thread Jacob Carlborg via Digitalmars-d-announce

On 2016-07-18 09:22, qznc wrote:


The hardest part is probably the work distribution. It should work
across platforms, so we can (eventually) test Windows, Android,
Raspberry Pi, etc.


GitLab can handle this really easy.


I don't believe GitLab would be a good idea. It is not built for this
and I find the CI parts quite minimal.


No, it's not built for this but I don't see a reason why it wouldn't 
work. I use GitLab extensively at work and it works great.



I have some buildbot experience, it would fit, but is a bitch to maintain.


GitLab is not :)

--
/Jacob Carlborg


Re: Autotesting dub packages with dmd nightly

2016-07-18 Thread qznc via Digitalmars-d-announce

On Monday, 18 July 2016 at 09:55:04 UTC, Sebastiaan Koppe wrote:
Not yet. Let me first do some groundwork. It could take month 
though.


I have a wife and two kids and I need to keep the scope limited.


Hey, me too. Slow and steady wins the race. ;)


Re: Autotesting dub packages with dmd nightly

2016-07-18 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Monday, 18 July 2016 at 07:22:07 UTC, qznc wrote:
Great! Maybe I can help you? Do you have a repository somewhere 
already?


Not yet. Let me first do some groundwork. It could take month 
though.


I don't think nightlies are that important. Older releases, 
alpha/beta versions, LDC, and GDC seem more important for me. 
For example, I would like to know if a dub package which was 
last updated two years ago is still working with the current 
dmd.


Once infrastructure is in place everything is just a git commit 
hash. Well, kind of.


The hardest part is probably the work distribution. It should 
work across platforms, so we can (eventually) test Windows, 
Android, Raspberry Pi, etc.


Like I said I am aiming really low. On purpose. I have a wife and 
two kids and I need to keep the scope limited. The first step is 
writing something that works in under 1kloc in 80% of the cases. 
If it would catch one regression per week/month before they end 
up in release I would be quite happy.


All the fancy stuff comes after that.

If we build something custom, the question is: Dogfooding or 
not? With (e.g.) Python we would have something working much 
quicker.


It should be written in D, that way everybody is a potential 
contributor.


Re: Autotesting dub packages with dmd nightly

2016-07-18 Thread qznc via Digitalmars-d-announce

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
Just to let you guys know - and to be sure no one is doing the 
same - I decided to go ahead and *start* writing an autotester 
that will fetch dmd nightly and unittest each dub package.


It will be using a classic master-worker architecture and will 
leverage docker containers.


I am aiming really low at first, but will eventually add things 
like memory usage, history, notifications, etc.


Great! Maybe I can help you? Do you have a repository somewhere 
already?


I don't think nightlies are that important. Older releases, 
alpha/beta versions, LDC, and GDC seem more important for me. For 
example, I would like to know if a dub package which was last 
updated two years ago is still working with the current dmd.


The hardest part is probably the work distribution. It should 
work across platforms, so we can (eventually) test Windows, 
Android, Raspberry Pi, etc.


I don't believe GitLab would be a good idea. It is not built for 
this and I find the CI parts quite minimal. I have some buildbot 
experience, it would fit, but is a bitch to maintain. Maybe v0.9 
is better when it is finished, but development happened at 
glacial speed in the last years (!).


If we build something custom, the question is: Dogfooding or not? 
With (e.g.) Python we would have something working much quicker.


Re: Autotesting dub packages with dmd nightly

2016-07-17 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Sunday, 17 July 2016 at 13:17:45 UTC, Jacob Carlborg wrote:

On 2016-07-16 22:34, Sebastiaan Koppe wrote:
Why not using something existing, like GitLab? Although GitLab 
is a source code hosting system its CI is excellent. It uses a 
master-worker architecture as well, GitLab being the master and 
one or more runners. Runners are available for all major 
operating systems: macOS, Linux and Windows. For Linux a Docker 
runner is available. This way it could run tests on multiple 
platforms. The installation is straight forward with native 
packages.


I don't have a good answer for this question. It might very well 
be the case that going with Gitlab (or similar) would be the 
better option.


Re: Autotesting dub packages with dmd nightly

2016-07-17 Thread Jacob Carlborg via Digitalmars-d-announce

On 2016-07-16 22:34, Sebastiaan Koppe wrote:

Just to let you guys know - and to be sure no one is doing the same - I
decided to go ahead and *start* writing an autotester that will fetch
dmd nightly and unittest each dub package.

It will be using a classic master-worker architecture and will leverage
docker containers.

I am aiming really low at first, but will eventually add things like
memory usage, history, notifications, etc.


Why not using something existing, like GitLab? Although GitLab is a 
source code hosting system its CI is excellent. It uses a master-worker 
architecture as well, GitLab being the master and one or more runners. 
Runners are available for all major operating systems: macOS, Linux and 
Windows. For Linux a Docker runner is available. This way it could run 
tests on multiple platforms. The installation is straight forward with 
native packages.


--
/Jacob Carlborg


Re: Autotesting dub packages with dmd nightly

2016-07-17 Thread Guillaume Piolat via Digitalmars-d-announce

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
Just to let you guys know - and to be sure no one is doing the 
same - I decided to go ahead and *start* writing an autotester 
that will fetch dmd nightly and unittest each dub package.


It will be using a classic master-worker architecture and will 
leverage docker containers.


I am aiming really low at first, but will eventually add things 
like memory usage, history, notifications, etc.


That would be really really great, especially if the unittests 
are also run in "release" build type.


Re: Autotesting dub packages with dmd nightly

2016-07-17 Thread Basile B. via Digitalmars-d-announce

On Sunday, 17 July 2016 at 06:19:16 UTC, Sebastiaan Koppe wrote:

On Sunday, 17 July 2016 at 04:47:40 UTC, Basile B. wrote:
I think that everybody will agree that's an excellent ideas to 
discover regressions. How do you plan to handle libraries that 
are not purely written in D (i.e requiring -L-lClib linker 
option) ? There are probably other cases where a build failure 
won't be significant.


Besides installing often used dependencies in the build image, 
I don't know. Lets see how many there are and go from there.


a new DUB property (in the package description) could solve this. 
This highly anticipated but in case your project becomes somewhat 
"official" there is this solution.


Re: Autotesting dub packages with dmd nightly

2016-07-17 Thread rikki cattermole via Digitalmars-d-announce

On 17/07/2016 6:15 PM, Sebastiaan Koppe wrote:

On Sunday, 17 July 2016 at 04:28:54 UTC, rikki cattermole wrote:

On 17/07/2016 8:34 AM, Sebastiaan Koppe wrote:
If you add nightly can you add x last major releases?


Yeah, specially for dub, nightly is not that important.


Also how about adding a 'button' for each one that says weather it
passed or not and for which version of dmd.


You mean like a badge? That is possible. Of course dub shows all
packages at once, so would have to coordinate to avoid a flood of requests.


Yeah badges. With caching that shouldn't be too much of an issue.
If you use redirection to the actual badge, it shouldn't eat too much of 
bandwidth or cpu time. After all, the badge should be the same for dmd 
version + pass/fail.


Re: Autotesting dub packages with dmd nightly

2016-07-17 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Sunday, 17 July 2016 at 04:47:40 UTC, Basile B. wrote:
I think that everybody will agree that's an excellent ideas to 
discover regressions. How do you plan to handle libraries that 
are not purely written in D (i.e requiring -L-lClib linker 
option) ? There are probably other cases where a build failure 
won't be significant.


Besides installing often used dependencies in the build image, I 
don't know. Lets see how many there are and go from there.


Re: Autotesting dub packages with dmd nightly

2016-07-17 Thread Sebastiaan Koppe via Digitalmars-d-announce

On Sunday, 17 July 2016 at 04:28:54 UTC, rikki cattermole wrote:

On 17/07/2016 8:34 AM, Sebastiaan Koppe wrote:
If you add nightly can you add x last major releases?


Yeah, specially for dub, nightly is not that important.

Also how about adding a 'button' for each one that says weather 
it passed or not and for which version of dmd.


You mean like a badge? That is possible. Of course dub shows all 
packages at once, so would have to coordinate to avoid a flood of 
requests.


Re: Autotesting dub packages with dmd nightly

2016-07-16 Thread Jack Stouffer via Digitalmars-d-announce

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
Just to let you guys know - and to be sure no one is doing the 
same - I decided to go ahead and *start* writing an autotester 
that will fetch dmd nightly and unittest each dub package.


It will be using a classic master-worker architecture and will 
leverage docker containers.


I am aiming really low at first, but will eventually add things 
like memory usage, history, notifications, etc.


Perhaps this code could also be used to find dub packages which 
are not currently compiling and mark them on code.dlang.org? 
Which is a feature which people have been asking for for a while.


Re: Autotesting dub packages with dmd nightly

2016-07-16 Thread Basile B. via Digitalmars-d-announce

On Saturday, 16 July 2016 at 20:34:49 UTC, Sebastiaan Koppe wrote:
Just to let you guys know - and to be sure no one is doing the 
same - I decided to go ahead and *start* writing an autotester 
that will fetch dmd nightly and unittest each dub package.


It will be using a classic master-worker architecture and will 
leverage docker containers.


I am aiming really low at first, but will eventually add things 
like memory usage, history, notifications, etc.


I think that everybody will agree that's an excellent ideas to 
discover regressions. How do you plan to handle libraries that 
are not purely written in D (i.e requiring -L-lClib linker 
option) ? There are probably other cases where a build failure 
won't be significant.


Re: Autotesting dub packages with dmd nightly

2016-07-16 Thread rikki cattermole via Digitalmars-d-announce

On 17/07/2016 8:34 AM, Sebastiaan Koppe wrote:

Just to let you guys know - and to be sure no one is doing the same - I
decided to go ahead and *start* writing an autotester that will fetch
dmd nightly and unittest each dub package.

It will be using a classic master-worker architecture and will leverage
docker containers.

I am aiming really low at first, but will eventually add things like
memory usage, history, notifications, etc.


If you add nightly can you add x last major releases?

Also how about adding a 'button' for each one that says weather it 
passed or not and for which version of dmd.