On 05/06/2019 22:11, Sam Tobin-Hochstadt wrote:
> There are a few things here:
>
Thanks for the clarification.
> 1. The set of tests is selected mostly based on what doesn't have a
> lot of dependencies, tests things in the `racket/racket` repository,
> and completes in the Travis CI time limit.
If there are more tests I should run please let me know. :) From what
you say the lists do not seem to be complete. Even if we do not run all
at every commit we could do it overnight - as we do for emulation
testing. Also, with GitlabCI we don't have a time limit anymore.
> 2. Some tests are run under `racket` vs `raco test` for historical
> reasons, probably that could be fixed. I originally took the command
> lines from the release checklist.
OK.
> 3. The tests are all together because building racket takes most of
> the time, and therefore splitting the tests didn't make sense. For a
> system with better pipeline support, splitting things up could work.
>
I now own a large server with 40cores and bucket loads of memory. In
total I now have about 70cores at home in servers for Racket CI.
Building racket3m takes about 3mins while building racketcs takes about
8mins with gcc -O3. I have the ability now to just run several tiny jobs
simultaneously which means I can have finer grained control over what's
passing and failing and mark it accordingly.
> One important background piece of information is that because
> drdr.racket-lang.org exists, typically other CI systems have focused
> on either system coverage or configuration coverage, and mostly on the
> core tests. Obviously that isn't ideal, but even before it tested
> RacketCS, DrDr took 8+ CPU hours to run a single full test run, and so
> doing that in other CI systems wasn't possible.
>
Now it is... :) What do you mean by a single full test run? All the
tests below, or running even further tests?
> Sam
>
> On Wed, Jun 5, 2019 at 3:49 PM 'Paulo Matos' via Racket Developers
> <racket-dev@googlegroups.com> wrote:
>>
>> Hi,
>>
>> I was looking at exploding the Racket CI configurations to make them
>> more fine-grained.
>>
>> For non x86_64 architectures with default flags / racketcs etc, there
>> are many failures and I would like to make the testing more fine-grained
>> so that I can lock a configuration /to success/ as soon as possible.
>> What I mean by this is... most our jobs at the moment have:
>> 'allow_failure: true'. I want to reduce these.
>>
>> Instead of having a testing job that does:
>> - raco test -l tests/racket/test 2>&1 | tee logs/test.log
>> - racket -l tests/racket/contract/all 2>&1 | tee logs/contract-test.log
>> - raco test -l tests/json/json 2>&1 | tee logs/json-test.log
>> - raco test -l tests/file/main 2>&1 | tee logs/file-test.log
>> - raco test -l tests/net/head 2>&1 | tee logs/net-head-test.log
>> - raco test -l tests/net/uri-codec 2>&1 | tee
>> logs/net-uri-codec-test.log
>> - raco test -l tests/net/url 2>&1 | tee logs/net-url-test.log
>> - raco test -l tests/net/url-port 2>&1 | tee logs/net-url-port-test.log
>> - raco test -l tests/net/encoders 2>&1 | tee logs/net-encoders-test.log
>> - raco test -l tests/openssl/basic 2>&1 | tee
>> logs/openssl-basic-test.log
>> - raco test -l tests/openssl/https 2>&1 | tee
>> logs/openssl-https-test.log
>> - raco test -l tests/match/main 2>&1 | tee logs/match-main-test.log
>> - raco test -l tests/zo-path 2>&1 | tee logs/zo-path-test.log
>> - raco test -l tests/xml/test 2>&1 | tee logs/xml-test.log
>> - raco test -l tests/db/all-tests 2>&1 | tee logs/db-test.log
>> - raco test -c tests/stxparse 2>&1 | tee logs/stxparse-test.log
>>
>> When one of them fails, we silence the failure with 'allow_failure:
>> true'. The problem with this is that if a problem shows up somewhere
>> else, it will go unnoticed because we allow failure by default. I want
>> to make each of these a single job and allow failure only on those which
>> really fail and once they pass, lock them onto success by not allowing
>> failure anymore.
>>
>> I initially copied the above lines without the redirections from travis
>> yml. Is this list of tests up-to-date? Is this all we have and are they
>> being run correctly?
>>
>> I should point out I find it slightly strange that the tests seem to be
>> implemented in different ways since the contract tests are ran with
>> `racket -l`, and stxparse are ran with `raco test -c`. What's the reason
>> for this difference?
>>
>> Kind regards,
>> --
>> Paulo Matos
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Racket Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to racket-dev+unsubscr...@googlegroups.com.
>> To post to this group, send email to racket-dev@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/racket-dev/a010e885-4f63-44d4-e73e-5457a0159fdc%40linki.tools.
>> For more options, visit https://groups.google.com/d/optout.
>
--
Paulo Matos
--
You received this message because you are subscribed to the Google Groups
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-dev+unsubscr...@googlegroups.com.
To post to this group, send email to racket-dev@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-dev/25e833d2-b792-8fc1-d5f6-e1446f282a46%40linki.tools.
For more options, visit https://groups.google.com/d/optout.
