#13950: Sage-patchbot improvements
-------------------------------+--------------------------------------------
Reporter: robertwb | Owner: mvngu
Type: enhancement | Status: needs_review
Priority: major | Milestone: sage-5.7
Component: doctest | Resolution:
Keywords: | Work issues:
Report Upstream: N/A | Reviewers:
Authors: | Merged in:
Dependencies: | Stopgaps:
-------------------------------+--------------------------------------------
Old description:
> Spkg up at
> http://sage.math.washington.edu/home/robertwb/patches/patchbot-1.2.2.spkg
> Most notably, this adds:
>
> * Startup-time plugin.
> * Non-ascii plugin.
> * Spkg inspection.
> * Better redundancy avoidance.
New description:
Spkg up at
http://sage.math.washington.edu/home/robertwb/patches/patchbot-1.2.3.spkg
Most notably, this adds:
* Startup-time plugin.
* Non-ascii plugin.
* Spkg inspection.
* Better redundancy avoidance.
--
Comment (by robertwb):
Replying to [comment:10 vbraun]:
> The even/odd values of k are main vs ticket?
> {{{
> Main: 0.65512 sec (25 samples, std_dev=0.00494)
> Ticket: 0.65906 sec (25 samples, std_dev=0.00382)
> }}}
> Just eyeballing things, they are within one standard deviation. So
Student-t is about 3. Which makes it pretty unlikely.
>
> I think its dangerous to remove the top 15 values here, though. It makes
the distribution artificially narrower, which inflates the t-value.
Really, what we're trying to do is estimate the minimum startup time, and
cut out the possibility of some short-term activity messing up the
timings. I'm open for other suggestions.
> Also, I was using the computer at the time and its possible that I did
something that was periodic with the frequency of the tests. E.g. click on
web links in a ~5 second rhythm. In other words, I'm not sure that it is
valid to treat the samples as statistically independent. Its probably
better to randomize which test counts as main and ticket for the startup.
I'd think it'd already be randomized based on when the test starts. I
alternate back and forth to try to eliminate any bias, but I suppose this
won't catch something periodic.
> {{{
> With 70% confidence, startup time increased by at least 0.5%
> With 75% confidence, startup time increased by at least 0.47%
> With 90% confidence, startup time increased by at least 0.36%
> With 95% confidence, startup time increased by at least 0.29%
> With 99% confidence, startup time increased by at least 0.17%
> With 99.7% confidence, startup time increased by at least 0.1%
> With 99.9% confidence, startup time increased by at least 0.035%
> }}}
> I'm a bit confused here. You can't be 100% sure that the startup time
increased at all. Which statistical test is going on? (Yes I could read
the source ;-)
Argh, found the error: https://github.com/robertwb/sage-
patchbot/blob/master/src/plugins.py#L244 I should be using variance, not
standard deviation. I'm using Welch's t-test (well, actually computing
various confidence intervals around the observed mean.)
Thanks for taking a deeper look at this. Wonder why it never tripped up on
my computer. Now I just hope we're doing enough samples to ever detect
anything...
--
Ticket URL: <http://trac.sagemath.org/sage_trac/ticket/13950#comment:11>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sage-trac?hl=en.
