Ken Brown via Cygwin-apps writes:
>>> - take an inordinate amount of time to run (exceeding the resource limits)
>>
>> That is a problem that comes with CI I think and we didn't really have
>> had to consider so far. I have a few packages that I don't run tests on
>> by default because the test suite produces hangs or other unstable
>> behaviour, but that is dealt with in the src_test function itself.
>> If there are really resource hungry tests they usually need to be
>> enabled somewhere and one could skip those if the build runs on CI (how
>> to find that out?).
>
> Here's an example where Jon's suggestion would have been useful: While
> building php recently, I noticed that the test suite took forever. I
> would have been happy to have a way to tell the CI to skip the tests
> and avoid exceeding the resource limits. But I wouldn't want to do
> that by modifying src_test, because I would still want to run the
> tests locally.
I think I didn't express myself clearly enough. If you look in the
perl-IO-Tty cygport file you'll find this:
src_test() {
cd ${B}
[ "no" != "${CYGPORT_RUN_UNSTABLE_TESTS-no}" ] &&
cygtest || echo "Unstable test, set CYGPORT_RUN_UNSTABLE_TESTS to run."
}
This is clearly an exceptional deficiency (even if presumably caused by
some border case in Cygwin) that needs to be handled in the package and
not a general thing that should be in cygport.
But that's not to say that something like CYGPORT_RUN_EXPENSIVE_TESTS
could not exist and arguably it should even be directly provided by
cygport itself if it was. Now, "expensive" has multiple dimensions, so
I'd expect that variable to have a list of values to indicate which of
these is considered relevant (like time, memory, network). There'd need
to be a matching set of values / limits provided by the local or CI
configuration to enable / disable tests as appropriate.
CPU wise the CI seems to have a pretty beefy configuration, I've not run
into memory problems yet (but didn't check that specifically). So at
the moment the relevant limits would be total runtime and maybe network
activity (I don't think I have a package that exercises that). But then
again, this could change with each run and certainly will change over
time, so hard-coding these limits doesn't seem like a good idea.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
