While I'm not against going back towards a checkout backwards that we can modify, I wonder if all the tests there should be there and how much do we actually duplicate.
Lucene 3x should include all of 3.0 tests + new ones that test new functionality, or assert bug fixes etc. There shouldn't be a test in 3.0 that does not exist in 3x, unless the missing test/feature was an agreed upon backwards break. So I think it would be really nice if backwards tested exactly what it should. For example, testing index format backcompat is done twice today, in "test-core" and "test-backwards", while it should only be run by backwards. There are a bunch of test classes I've created once that impl/extend 'search' related classes, for back-compat compilation only. They should also be run in backwards only. The downside of this is that maintenance is going to be difficult - it's much easier to copy tests over to backwards, then decide which ones should go there and which shouldn't. Also, adding new API requires a matching backwards test etc. Not non doable, but difficult - requires discipline. Shai On Tue, Nov 30, 2010 at 2:02 PM, Robert Muir <[email protected]> wrote: > On Tue, Nov 30, 2010 at 4:47 AM, Shai Erera <[email protected]> wrote: > > > > Like you said, the rest of the tests just increase the test running time. > > > > I'm not completely sure about this: do we always switch over our tests > to do the equivalent checks both against the new API and the old API > when we make API changes? There could be bugs in our 'backwards > handling' that are actually logic bugs that the new tests dont detect. > > So I'm a little concerned about only running "pure" simplistic API > tests in backwards. > > On the other hand, I'm really worried about what Shai brings up here: > we are doing some refactoring of the tests system and there is more > shared code at the moment: similar to MockRAMDirectory. > Because we worry about preventing things like index corruption, its my > opinion we need things like MockRAMDirectory, and they should be able > to break all the rules/etc (use pkg-private APIS) if we can prevent > bugs. > Just look at our trunk or 3.x tests and imagine them as backwards > tests... these sort of "utilities" like RandomIndexWriter will be more > fragile to internal/experimental/pkg-private changes, but as mentioned > above i think these are good to have in backwards tests. > > So, I think at the moment I'm leaning towards the idea of going back > towards a checkout that we can modify, in combination with us all > soliciting more reviews / longer time for any backports to stable > branches that require backwards tests modifications? I understand > Uwe's point too, its dangerous to modify the code and seems to defeat > the purpose of backwards, but I think this is going to be a more > serious problem after releasing 3.1! > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
