2006/11/21, Gregory Shimansky <[EMAIL PROTECTED]>:
Geir Magnusson Jr. wrote:
>
>
> Alexey Varlamov wrote:
>> Folks,
>>
>> I've just did a little step in improving pre-commit testing for DRLVM;
>> most important change is a move from "perTest" forking mode to "once"
>> (aka sameVM mode).
>> This reduces testing time drastically (~50%), but may introduce some
>> extra instability (like new intermittent failures or timeouts).
>
> Plus the inability to figure out what's screwing things up.
>
> I can't decide if I like this. On one hand, I like it because it's
> actually better to find side-effects - some test can pass, but still
> leave the VM in a broken state that another test will show.
I think the change has a good intension but it should've gone through
more testing. Preferably the fixes for problems on windows should've
been included into the patch. If the problems are too hard to fix, why
not open a discussion before changing the files in SVN?
I did test on SUSE9, Win2003 server and WinXP(laptop) - only spotted
*intermittent* failures on WinXP. I assumed it maybe little bothering
but really easy to switch; OTOH we better discover instabilities
sooner if they are frequent.
> OTOH, it does remove the clarity of each test being a single, separate
> test. It conceptually mixes integration testing with unit testing.
>
> Can you please just add a switch? That way in the event of a failure,
> we can re-run with forking on, and therefore can tell if the crash is
> specifically due to the test that is crashing, or a side effect caused
> by something that came before.
I would also like a switch. I don't like it that a change which
knowingly introduces problems with acceptance tests is committed without
discussion. Should we discard kernel tests from drlvm commits until they
are fixed? I think that there was an agreement about no regression. So
now all commits to drlvm are blocked.
But there *is* cmd-line switch, my bad that I didn't explained this
more clearly. And I'm not sure we firmly adopted "zero regression"
policy - at least intermittent failures in kernel tests did block
nobody so far. Should we vote formally, BTW?
Generally I agree it is always better to discuss, and in fact
hesitated to commit with default forking set to "once" - but decided
this is right thing to do rigth now. The current trend to make drlvm
truly standard VM for harmony, recently Alexei even proclaimed 100%
HUT pass rate, neverthless drlvm is not yet reliable enough and we
should keep the momentum and face possible problems with open eyes.
Again, I'm sorry to bother you^H^H^H us VM guys, but believe this is
neccessary evil now.
--
Alexey
