David has indeed made some very hepful contributions to this -- I remember the cremona_letter_code business well (the function just assumed that the input was valid), and there was another case to do with validating the input to mwrank, an external C++ program included in Sage, which took a bit of work but ended up much better than it had been.
For quite a lot of Sage's number theory functionality many of us do regularly test against Magma, and this has in the past revealed bugs on both sides. To a lesser extent tests are possible against the pari library, but of course that tells you little when the only method implemented in Sage involvers a call to the pari library. Many contributors to Sage have only had experience of writing code for them,selves before, and there is a lot to learn about in the different world of open source sofware where you cannot assume that the person running your code has a PhD in the area.... John On 13 May 2014 13:22, Dr. David Kirkby <[email protected]> wrote: > On 13 May 2014 10:48, <[email protected]> wrote: > > Hello, > > > > are here developers in the group who actively test (the routines they > use) > > e.g. by throwing random input at them? > > (like in ticket http://trac.sagemath.org/ticket/1136 ) > > > > If yes, which other techniques do you use to detect bugs? > > > > Thanks, > > I have done a couple of related things, as I have sometimes had > concerns about the Sage testing - I think more so in the past than > now. Some of the code was of very dubious quality in my opinion. I > recall an email from one developer telling me he did not see much > point in testing much, as he would just fix bugs when reported !!! > > I have several times advocated the use of Sage releases where no new > functionality is added - only bugs were fixed. There has been little > appetite among developers for this. Anyway, a couple of things I have > done. > > #1 > > I developed some code to send random data to commands and uncovered > some which don't behave as expected. > > cremona_letter_code > > for example should only work for non-negative integers, but I found that > > sage: cremona_letter_code(23.23) # Invalid input, gives output. > 'x' > sage: cremona_letter_code(-3) # Invalid input, hangs using lots of CPU > time. > > That was > > http://trac.sagemath.org/ticket/10105 > > I found a number of examples of functions which could mis-behave if > fed invalid input. > > #2 > Some time ago I suggested that we could compare Sage output with > output from WolframAlpha to use as a note in doc tests, giving the > WolframAlpha URL that gives the same result. I felt WolframAlpha was > preferable to Mathematica, since everyone had access to it. Alex > Ghitza in particular thought we might be breaking the terms of the > license, and on reading it, he could well have had a valid point. So I > asked Wolfram Reseach to clarify the license, and if we were permitted > to use WolrramAlpha to make comparisons with Sage. They basically said > yes, although we can't make automated querying of their servers. That > is hardly surprising, as calling their servers thousands or millions > of times would load their servers. You can read their response here. > > https://groups.google.com/forum/#!topic/sage-devel/tijz3svNNiE > > I don't know if anyone has actually done this, making notes in doc > tests of the WolframAlpha / Mathematica output. But it would give some > confidence that results are the same. > > I do recall one function, related to prime testing, where Mathematica > was a lot faster than Sage. On closer inspection, it was revealed that > the Mathematica algorithm had been proved write for N < $some_value, > but there was no proof it worked for all values, whereas the Sage test > was more thorough. I reported this to Wolfram Research, who said they > would amend the documentation to reflect this. I can't recall whether > they did. > > I think there is some value to comparing Sage and Mathematica output > for tests, and perhaps also noting the CPU time used. If Sage is > significantly slower, it might indicate there are better algorithms in > existence, and those could be investigated. > > > Dr. David Kirkby G8WRB > http://www.vnacalibration.co.uk/ > Economical & accurate VNA calibration kits. > Coefficients available for HP, Agilent, Anritsu, Rohde & Schwarz and > VNWA network analyzers. > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
