On Fri, Aug 16, 2013 at 3:37 PM, Regina Henschel <rb.hensc...@t-online.de> wrote: > Hi Rob, > > Rob Weir schrieb: > >> Moving this topic to its own thread. >> >> It should be possible to code a very thorough set of test cases in a >> spreadsheet, without using macros or anything fancy. Just careful >> reading of the ODF 1.2 specification and simple spreadsheet logic. >> > > Following the spec is not enough. For example, if the accuracy decreases > from 14 digits to 7 digits, that is not covered by the spec. > > <skip test case example description> >
This is not hard to check. When you save an ODF document in version N, it saves the last-calculated value of each cell as well. It would be possible to verify that the same values are returned in AOO version N as well. Probably not via AOO itself, but you can access the last-saved value in the cell via the ODF Toolkit, for example. Load sheet, save, compare to sheet saved in previous version (or some other reference version). > >> If we used an approach like this on the other spreadsheet functions, >> we could have a semi-automated test suite that would practically >> guarantee that Calc is free of calculations errors. Once we're >> written the test cases, a modest upfront investment > > > "modest"? One function a day and you need more than a year. > There are ways of making this more efficient. You group together functions that are related and have the same basic input values. For example, the many bin/oct/hex conversion functions, the logical IF/AND/XOR functions, the DCOUNT/DMAX/DMIN database functions, etc. Try to reuse the same test conditions for related functions. So there is a lot of reuse possible. But the parts are also very independent, so the work can be done in parallel by any interested volunteers. > > , it will benefit >> >> us with every release we do. Heck, it would benefit LibreOffice, >> Gnumeric, Calligra as well, maybe even Microsoft and Google, though >> they might already have such test cases defined internally. > > > I see a problem in how such a test suite is made available. And how the > results for a special release are collected. > I was thinking of storing them in SVN, something like /test-docs/calc/functions or something like that. I think we want to make it easy for a tester to download the whole collection of test sheets without needing to download them individually from a wiki page, for example. > The problem with the current test cases is, that I do not know where they > are, how they are to use and how to generate new ones. It is a closed book, > only for insiders. > Do we have test documents like this today? I don't want to do redundant work. > >> >> Anyone interesting in helping with this kind of test case development? > > > There exist some files already in Bugzilla. I used to make test documents, > when working on functions. I think, that they can be extended to work in a > way, that a simple look on it will tell errors. But I have no ready > collection on my PC and most will be already deleted from my PC in the > meantime. > > One problem is, that comparisons with constants have to be written in a way, > that they are independent from local. Eike has once corrected one of my test > spreadsheets that way. > Or we agree on a locale to be used for calculation testing. There are only a small number of locale-dependent calculations in Calc. I have another test document that triggers all the implementation-dependent, implementation-defined, locale-defined and undefined behaviors in OpenFormula. For those there is no single "correct" answer. But we can use it to verify that the calculations do not change unexpectedly from release to release. Or put differently, it might make sense to isolate the locale-sensitive calculations from the locale-insensitive ones and approach the testing of them differently. > >> >> Any ideas on how to fully automate this? ODF 1.2 is very strict, so >> we're not starting from a perfect score. But we should find an easy >> way to report on regressions. > > > If you will automate this, you will need to develop a frame. But automation > is not the total solution. Testing can be a way to bring user into the > community. And tests have to cover different languages and scripts. I > remember errors reported to LibreOffice, where a time calculation was wrong > only in special locals. To extend a testing frame to consider this would be > very expensive. > > Let me not be misunderstood, I like the idea of collecting test cases. > I'm thinking less of "collecting" test cases and more of designing a test suite with specific coverage goals in mind. The coverage may not be 100% if you consider the locale aspect of it. But it is 100% within the domain it aims to cover, and by doing that insulates from defects in that area. >From a community perspective I think it would have value attracting those who are more interested in the formal aspects of QA and in building skills there. But it would not be interesting to everyone. Regards, -Rob > Kind regard > Regina > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org > For additional commands, e-mail: dev-h...@openoffice.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
