And this design has problems e.g. https://trac.sagemath.org/ticket/28599
Basic story is: The random splitting that is being done to find real roots is far from stable. Apparently the doctests don't catch on, because they always use the same "random state". In real life, you cannot obtain a regular polygon with 20 vertices with this method. So some doctests claim that sage can do something, which can be totally wrong (unless you have your lucky day). Maybe if the doctests would have shown some variation, people designing this would have caught on that their random choice could be real bad and they would have to accounted for that. Am Montag, 22. Juni 2020 13:43:39 UTC+2 schrieb Dima Pasechnik: > > > > On Mon, 22 Jun 2020, 12:35 'Jonathan Kliem' via sage-devel, < > sage-...@googlegroups.com <javascript:>> wrote: > >> Yes there is need. Please go into any file of your choice and print out a >> "random" value in a doctest and see what happens. >> > > yes, this is by design. Otherwise one cannot get reproducible doctests. > To get more randomness you need to pull it out of the random source. > Randomness is about distributions, not particular values (sorry, I start > to sound like I teach a statistics class :-)) > > > There is not even a variation as Michael pointed out. >> >> If you start sage in a shell, there is variation. But sage in a shell >> behaves different than in doctests. E.g. in doctests the dictionaries are >> being sorted, in the shell not. >> And currently in doctests there is no variation at all in random numbers. >> You will always get the very same numbers. >> >> Am Montag, 22. Juni 2020 13:26:24 UTC+2 schrieb Dima Pasechnik: >>> >>> >>> >>> On Mon, 22 Jun 2020, 12:18 'Jonathan Kliem' via sage-devel, < >>> sage-...@googlegroups.com> wrote: >>> >>>> Currently, the doctest setup produces the very same random numbers on >>>> every instance. >>>> >>>> If you call `set_random_seed()` first you will get somewhat different >>>> values on every single doctest run. It can still be annoying as not every >>>> bot will catch every failure, but it is by far more likely to catch >>>> regression eventually. >>>> >>> >>> there is no need to explicitly set the seed in these tests. Because it >>> is done as Sage starts up. >>> By explicitly setting it you just restart the RNG, for no good reason. >>> There is no guarantee that this would give you "better" randomness. >>> If you want more random data in your doctest, repeat tests in a loop. >>> >>> >>>> It seems that there are a number of doctests that are supposed to run >>>> on different values each time the doctest is run. However, currently >>>> exactly one list of values is tested each time. >>>> >>>> It's not about getting very good pseudo randomness. It is just about >>>> getting some variation of values. >>>> >>>> Am Montag, 22. Juni 2020 13:09:55 UTC+2 schrieb Dima Pasechnik: >>>>> >>>>> >>>>> >>>>> On Mon, 22 Jun 2020, 10:58 'Jonathan Kliem' via sage-devel, < >>>>> sage-...@googlegroups.com> wrote: >>>>> >>>>>> I had a doctest >>>>>> >>>>>> sage: ls = [randint(-100,100) for _ in range(4)] >>>>>> sage: intervals = [[x, x+randint(1,50)] for x in ls] >>>>>> sage: P = polytopes.hypercube(4, intervals, backend= >>>>>> 'field') >>>>>> >>>>>> which then checked `P` for consistency. However it was somewhat >>>>>> meaningless (missing `set_random_seed()`) as all entries in `ls` where >>>>>> positive. It never caught a sign error that we introduced in #28866 >>>>>> <https://trac.sagemath.org/ticket/28866>. >>>>>> Adding `set_random_seed` to it, does catch the error. >>>>>> >>>>> >>>>> unless I misunderstand how Sage random numbers are set, by adding an >>>>> explicit set_random_seed() call you merely went with another bunch of >>>>> pseudorandom values than these ones you were hitting without it, not that >>>>> you were fixing a bug. In other words you just hit a more favourable >>>>> bunch >>>>> of pseudorandom data. >>>>> >>>>> To make such tests more robust, one needs to call them in a loop, >>>>> without resetting the seed. Then you get a bit more of (pseudo) >>>>> randomness. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Now this check for consistency certainly wasn't very good in the >>>>>> first place, as two polyhedra might compare equal, even if one is set up >>>>>> incorrectly (I just noticed this because of your post, thank you). But >>>>>> it >>>>>> still would have caught the error even randomness was true randomness. >>>>>> >>>>>> You might say, I should have known to add `set_random_seed()` in the >>>>>> first place, but I'm definitely not the only one who made that mistake >>>>>> and >>>>>> there are plenty of doctests that claim to test a random event, but they >>>>>> really test the same event all over and over. >>>>>> >>>>>> Am Montag, 22. Juni 2020 11:24:59 UTC+2 schrieb Dima Pasechnik: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, 22 Jun 2020, 10:01 'Jonathan Kliem' via sage-devel, < >>>>>>> sage-...@googlegroups.com> wrote: >>>>>>> >>>>>>>> Hi. Could we maybe turn off reproducible "randomness" in doctests >>>>>>>> by default. There are probably thousands of meaningless doctests >>>>>>>> because of >>>>>>>> this. >>>>>>>> >>>>>>>> E.g. https://trac.sagemath.org/ticket/29904 >>>>>>>> >>>>>>> >>>>>>> I don't understand, you say there that it went unnoticed due to a >>>>>>> too easy doctest. I don't see how that one had anything to do with >>>>>>> randomness. >>>>>>> >>>>>>> Also, could you explain what you mean by "randomness" there. >>>>>>> Doctests are either tagged "random" (and do matching of answers >>>>>>> skipped), >>>>>>> >>>>>>> or use random inputs. If a doctest with random input produces >>>>>>> inconsistent results due to not properly set seeds, it would have been >>>>>>> caught all along. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> or the doctests for the namespace function `bernoulli` or the >>>>>>>> `number_of_partitions` function also in the namespace. >>>>>>>> >>>>>>>> Alternatively we should add `set_random_seed` to all of those >>>>>>>> places. >>>>>>>> >>>>>>>> Either way, this might be a huge project, because of possible bugs. >>>>>>>> >>>>>>>> Jonathan >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "sage-devel" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to sage-...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/sage-devel/6a5005ec-717e-400c-b253-514f9101bc72o%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/sage-devel/6a5005ec-717e-400c-b253-514f9101bc72o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "sage-devel" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to sage-...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/sage-devel/fcc3910a-e851-4f58-9a26-4eaf3dced699o%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/sage-devel/fcc3910a-e851-4f58-9a26-4eaf3dced699o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "sage-devel" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to sage-...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/sage-devel/f5004f9e-be47-47a2-9646-6e3eaf6ac1b6o%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/sage-devel/f5004f9e-be47-47a2-9646-6e3eaf6ac1b6o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "sage-devel" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sage-...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/sage-devel/ac071eda-2f54-45a2-9434-5104341dcb36o%40googlegroups.com >> >> <https://groups.google.com/d/msgid/sage-devel/ac071eda-2f54-45a2-9434-5104341dcb36o%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/bdd1663f-9898-417a-890f-84d4ed25425bo%40googlegroups.com.
