[email protected] said:
> > Random question of the day: do you happen to have stats about how
> > often those insignificant tests actually fail? That should help to
> > figure out which ones are actually working, and therefore should not
> > be marked as insignificant.
>
> I'm glad you asked. I spent some time yesterday and today doing that
> analysis for the 40-odd insignificant tests that had no Jira task associated.
>
> I used the data from the last eleven CI runs for each module (times eight
> configs for which autotests are executed). That's not hugely statistically
> significant, but it's not that easy to pull large numbers of build logs out
> of the CI system manually through the web interface. (Any CI folks who can
> tell me how to extract/search all the stored logs?)
I'm using this pattern (which has some minor problems not relevant for
any test I've looked at so far):
from=02238
to=02538
prj=QtBase_master_Integration
cfg=win32-msvc2010_Windows_7
test=tst_QLocale
trailing_lines=3
for i in $(seq -w $from $to); do
echo "$i:"
wget --quiet -O -
http://testresults.qt-project.org/ci/$prj/build_$i/$cfg/log.txt.gz |
zcat |
egrep -A $trailing_lines "Testing $test" ||
echo "did not run"
echo ""
done 2>&1 | tee results_${test}.txt
... and scanning at least one hundred logs.
Having a transparent caching proxy between self and
testresults.qt-project.org helps a lot.
_______________________________________________
Development mailing list
[email protected]
http://lists.qt-project.org/mailman/listinfo/development
