On Fri, Jun 20, 2003 at 10:04:09PM -0700, Dann Corbit wrote: > > On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote: > > > > Citing Tom Lane: > > > > I have been through crash-me in some detail, and it left a > > > > very bad taste in my mouth. Don't bother holding it up as an > > > > example of good practice. > > > > > > Every single test in their list is interesting and useful. > > > > At least on the version I just saw there are several results > > with Postgres that are weird (table names > 500 chars?). > > It does get silly at a point, but I have seen systems with 128 > characters for table names, column names, etc. Some people seem to like > it. Not me. Too much typing.
I meant that the real limit on 7.2 was much lower than that unless they twiddled with sources at compile time (there's no configure switch for that AFAIR). Maybe 31 or 63 chars, I don't remember. Do you really trust the rest of the test seeing that they came up with a clearly wrong answer in such a simple test? They can't even "make vacuum run reliably" on 7.1. See the performance test. Maybe they want to test 7.3 with lazy vacuum in place. Why don't they do that? 7.1 is already 2 years old. > > Other things tested are clearly wrong (things that are = > > NULL, > > Sounds like testing for the existence of a bug. > X = NULL > X <= NULL > X >= NULL > Etc. must always test false, regardless of the contents of X. Test for > equality with NULL is a conformance error if NULL == NULL returns true. You see, you are saying "sounds like they are testing". What does the code actually test? Which is the right behaviour? Which behaviour gets the green point, MySQL's or the right one? There are lots of things like this; I don't want to waste my time actually reading the code to see what the correct answer for each test is. About the 1--1 thing Tom mentioned: be aware that Postgres happily accepts the correct 1 - -1 expression, but also correctly fails to "calculate" 1--1. Which one gets the green point? Of course it's the non-compliant one. Also they don't test things they don't support. Is there a test for subselects? What about concurrency? Transactional issues? What about performance when they have their "transaction support" enabled? > So far, I have seen three problems pointed out (out of 600+ tests). > That's 0.5% defects. Why not just drop the stupid tests, or bend them > to test for what they ought to be testing. There's already a mechanism for testing inside Postgres. Maybe more tests are needed, but crash-me offers no real value. I just became aware that the NIST test suite is quite old. Maybe what's needed is to expand it to SQL3 to develop a way of measuring the compliance level. But the cost of doing that is probably prohibitive. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans) ---------------------------(end of broadcast)--------------------------- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html