Re: Adding more projects to the Project Tester
On Monday, 9 July 2018 02:44:31 MDT Seb via Digitalmars-d wrote: > On Monday, 9 July 2018 at 05:58:17 UTC, Jonathan M Davis wrote: > > then ./test.sh will run the tests without -release, and > > ./testRelease.sh will run the tests with -release (simply > > running dub test won't do anything useful, because all of the > > unittest blocks are versioned to avoid problems when projects > > depending on dxml compile their tests). > > The Project Tester will by default look at (if existent) the > .travis.yml and run the same 'script' commands, so it should do > the right thing, but I will keep this in mind. Thanks for the > explanation! Ah. Then I will have to figure out what needs to go in .travis.yml and add it. - Jonathan M Davis
Re: Adding more projects to the Project Tester
On Monday, 9 July 2018 at 05:58:17 UTC, Jonathan M Davis wrote: On Thursday, 5 July 2018 21:19:44 MDT Seb via Digitalmars-d wrote: [...] dxml should probably be on the list. Yes! https://github.com/dlang/ci/pull/230 Assuming that this is only on *nix and not Windows Yes. (I have yet to write a test script for Windows) This might help: https://github.com/Abscissa/AppVeyor-D then ./test.sh will run the tests without -release, and ./testRelease.sh will run the tests with -release (simply running dub test won't do anything useful, because all of the unittest blocks are versioned to avoid problems when projects depending on dxml compile their tests). The Project Tester will by default look at (if existent) the .travis.yml and run the same 'script' commands, so it should do the right thing, but I will keep this in mind. Thanks for the explanation!
Re: Adding more projects to the Project Tester
On Thursday, 5 July 2018 21:19:44 MDT Seb via Digitalmars-d wrote: > So learning from the recent Vibe.d regression fiasco (we > temporarily disabled a subconfiguration in Vibe.d and promptly > got a regression in 2.081), I think we should try to add more > projects to the Project Tester. > > The current list is here: > https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 > > Any suggestions? > > Why should I add my project to the Project Tester? > -- > > Once a project is added to the Project Tester, DMD can't regress > on it anymore as for every PR at dmd, druntime, phobos, tools and > dub the testsuite of the added projects are run. > > How does the Project Tester work? > - > > - By default, it will run the same commands as Travis would do. > Although, if necessary, custom commands can be used too. > - It will checkout the latest, stable git tag > > Requirements > > > - moderately popular or was prone to regressions in the past > - rather easy to build (i.e. you don't need to download and > recompile clang) > - no flaky testsuite (random errors in the testsuite due to > network connectivity shouldn't happen. Though there's > `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of > your testsuite) > - reachable author or development (if there's ever a case where > we need to push changes via a trivial PR to the repo, it > shouldn't sit in the queue for weeks) dxml should probably be on the list. Assuming that this is only on *nix and not Windows (I have yet to write a test script for Windows), then ./test.sh will run the tests without -release, and ./testRelease.sh will run the tests with -release (simply running dub test won't do anything useful, because all of the unittest blocks are versioned to avoid problems when projects depending on dxml compile their tests). - Jonathan M Davis
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 05:02:56 UTC, Joakim wrote: The LDC compiler? kinke recently had an issue because of all the C++ integration changes upstream: https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813 As perhaps the largest consumer of extern(C++), it may make sense to add it for all the C++ work being done. It would require the llvm package be pre-installed in the test environ. List of current projects looks great, was tough to think of anything to add. Good idea. Not sure how easy/hard it is, but let's give it a try: https://github.com/dlang/ci/pull/228
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 21:25:28 UTC, tide wrote: On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote: So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? - - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks) Include Windows as part of the testing done. Just to avoid confusion, testing of DMD/Druntime/Phobos/etc. is done on Windows: https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32 https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32_64 https://ci.appveyor.com/project/greenify/dmd ... In my experience are the huge majority of all regressions originating in the D frontend where architecture actually doesn't matter so much. That being said, we are currently investigating to move the Project Tester from Jenkins to Buildkite (https://github.com/dlang/ci/pull/225) and one advantage of this move is that it'll be a lot easier to add new machines to the build cloud (so you could then hook up that Windows machine in your garage with the Project Tester). However, AFAICT many of the dub projects do have dependencies that are harder to install on Windows, so this might not happen in the near future.
Re: Adding more projects to the Project Tester
On Saturday, 7 July 2018 at 08:04:47 UTC, Timoses wrote:
On Friday, 6 July 2018 at 23:56:01 UTC, Basile B. wrote:
On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:
By the way, is there any policy for outdated dub packages?
You just found an idea for the score algorithm.
Why isn't there something like "compiler compatibility" in a
dub config file? E.g. currently the vibe.d project lists the
compiler versions the code is compatible with [1].
Wouldn't a field in dub like
"compilerCompatibility":
{
"dmd": "2.080.0",
"ldc": "..."
}
be nice? This could also help code.dlang.org to show
compatibility of libraries/packages.
The only pain I guess would be to keep such a list up to date.
[1]: https://github.com/vibe-d/vibe.d#support
I'm not sure it's a good idea to manually hard-code such a list
into the dub.json.
However, for every release or so, the dub registry could crawl
all its packages and run `dub test` on them. Of course, there's
the chance that a build still fails (e.g. due to missing
dependencies) even though it would pass locally, but each package
that passes with the latest registry could receive additional
points.
Re: Adding more projects to the Project Tester
On Saturday, 7 July 2018 at 16:29:15 UTC, Guillaume Piolat wrote: Please include the intel-intrinsics package, its test expose a regression on 2.081 that I haven't looked after yet. The goal is to provide a _stable and usable_ SIMD interface. https://github.com/dlang/ci/pull/224
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote: So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. A meta-problem is that many package unittest (at least on typical CI) don't exercise the DMD optimizer since they don't run with -O or -release. Dub has no default options to dub test with optimizations so ti's very rarely tested.
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote: Requirements - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks) Please include the intel-intrinsics package, its test expose a regression on 2.081 that I haven't looked after yet. The goal is to provide a _stable and usable_ SIMD interface.
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 23:56:01 UTC, Basile B. wrote:
On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:
By the way, is there any policy for outdated dub packages?
You just found an idea for the score algorithm.
Why isn't there something like "compiler compatibility" in a dub
config file? E.g. currently the vibe.d project lists the compiler
versions the code is compatible with [1].
Wouldn't a field in dub like
"compilerCompatibility":
{
"dmd": "2.080.0",
"ldc": "..."
}
be nice? This could also help code.dlang.org to show
compatibility of libraries/packages.
The only pain I guess would be to keep such a list up to date.
[1]: https://github.com/vibe-d/vibe.d#support
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote: On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote: Why should I add my project to the Project Tester? -- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. By the way, is there any policy for outdated dub packages? You just found an idea for the score algorithm.
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote: Why should I add my project to the Project Tester? -- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. By the way, is there any policy for outdated dub packages? There are some packages in dub registry, which don't build with the latest D version anymore. Should such packages stay in the registry? It gives a bit inflated view of packages, seems like there are many, but 50% (for example) aren't even usable without additional effort.
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote: So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? - - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks) Include Windows as part of the testing done.
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 05:02:56 UTC, Joakim wrote: The LDC compiler? kinke recently had an issue because of all the C++ integration changes upstream: https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813 As perhaps the largest consumer of extern(C++), it may make sense to add it for all the C++ work being done. It would require the llvm package be pre-installed in the test environ. List of current projects looks great, was tough to think of anything to add. Thats a great idea! Especially since there is a whole lot more extern(C++) stuff coming in the next DMD release.
Re: Adding more projects to the Project Tester
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote: So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? - - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks) The LDC compiler? kinke recently had an issue because of all the C++ integration changes upstream: https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813 As perhaps the largest consumer of extern(C++), it may make sense to add it for all the C++ work being done. It would require the llvm package be pre-installed in the test environ. List of current projects looks great, was tough to think of anything to add.
Adding more projects to the Project Tester
So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? - - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks)
