Hi,
Jörg Jahnke schrieb:
There seem to be timing (or similar) problems with some of the test -
which is a bad thing. It just takes time to find and fix these
problems. OTOH there is a set of tests, the Cat0-tests, which should
run just fine and should be reproducable on most if not any system,
because these tests were checked and fixed.
Andre: Did you try to run the Cat0-tests alone and see whether the
results are the same as those from the tests done here in Hamburg?
If my test results differ from those in Hamburg, this is what I usually do:
- re-run the failed tests manually
- if these tests still fail, I start to debug the testscripts, running
just the failing parts and / or verify the failed parts manually
- If I am able to lokate the error I file an issue
Problem is the last step in many cases. It happens often, that erors go
away when I start debugging. In this case I ignore the error.
For a very long time I have been testing mainly on Windows / Linux and
got almost the same results as the Hamburg QA team got. Interesting
about this is, that I did the tests in a "polluted" environment (Windows
tests ran within a regular user setup ometimes with old settings. Linux
testing was done on remote XServer).
Within the last weeks I started to do some other tests. These have been
comparision tests on fresh Linux VM (Kubuntu 7.10 using KDE 4.2 not
*that* unusual system I'd guess). The tests have been done to compare a
OOo "Sun build" vs. buildbot and the second was done to compare vanilla
OOo to distribution version of OOo.
The runs did no reveal many differences between the builds - but had a
lot of errors compared to the Sun test results. I did not much analyzing
on the results, as it was just for comparision and this was more or less
achieved.
The other tests I did have been on Solaris Sparc. I did quite the same
as I did with the Linux tests: remote testing using a cygwin X-Server on
windows.Those testresults are now in Quaste (OOo310m9, Solaris Sparc,
German). Unfortunately I have not the chance to do deeper analysis of
these results.
The first test runs on this machine had been done using the testtool
that ships with OOo. After some remarks from Helge fixed my script and
did a second run with the "official" testtool. But this did not change
the results. RE-Running the failed tests manually did also not changa
the result. (
As said - I had not change to do more analysing).
Developers also might have the expectation that they can use
testautomation in their environment. But this is also not possible
for the same reasons.
For the Cat0-tests this would be possible.
I'd rather say "should in theory". See above and look at Quaste - even
the cat0 tests are not yet ready and reliable enough for this.
best,
André
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
