I think half of your idea is good: we should run the tests not on the
skip list, and expect them to fail. That'll force us to remove them
from the skip list as they become applicable (we're catching up with
JS 1.7/1.8 features, so those tests are expected to fail less and
less). I don't think we should be switching to an explicit list of
tests to run; as long as we adopt the idea of running the tests on the
skip list with expectation to fail. Then it doesn't matter which list
do we maintain, and the skip list is definitely smaller...
Attila.
On 2008.10.21., at 16:17, Marc Guillemot wrote:
Investigating a bit more the StandardTests, I find that 7832 tests
pass
if I ignore base1.skip and opt1.skip whereas only ~6542 pass when skip
files are used. This means that ~1300 tests that previously didn't
pass
(or didn't exist) are now green, that's a good news. On the other
side,
this means as well that they may become red again without notice as
they
are not currently considered.
As a consequence, I propose following:
- build a "positive" list of tests that pass
- run *all* the standard tests from the tests dir
- report a failure if a test from the positive list fails
- report a failure if a test that is not in the positive list passes
I believe that it is better to work with a positive list because the
tests set is not under the control of the Rhino committers and I guess
that newly added tests are more likely to fail than to pass (new
language features, ...).
Tests that depends on OS or computer capacities still need to be
handled
separately.
Thoughts?
Cheers,
Marc.
--
Web: http://www.efficient-webtesting.com
Blog: http://mguillem.wordpress.com
Marc Guillemot wrote:
Norris Boyd wrote:
...
Which tests are you referring to? I run the tests that are shared
with
SpiderMonkey using org.mozilla.javascript.drivers.JsDriver, which
has
skip lists, see http://developer.mozilla.org/En/Running_the_Rhino_tests
.
Those tests all pass for me, although I need to resync to pull down
any new tests that have been introduced on the SpiderMonkey side.
How are you running tests? Using
org.mozilla.javascript.drivers.StandardTests?
Exactly these tests. Running it with ant junit-all, I have most of
the
time 98 failures (out of 6641 tests). On the server with
CruiseControl,
I have more failures but I guess that this is only because it is a
bit
more slow. This is made with the latest js tests from CVS head.
When I run ant jsdriver-run this fails as well, but no idea how much
failures occur then.
If I compile and run the tests in testsrc/org/mozilla/javascript/
tests
(which are not included in any Ant target) from my IDE, I have 14
errors
(12 of them in DoctestsTest).
Cheers,
Marc.
_______________________________________________
dev-tech-js-engine-rhino mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-tech-js-engine-rhino
