On 08.10.2021 20:18, Will Godfrey wrote: > On Fri, 8 Oct 2021 19:44:48 +0200 > Ichthyostega <p...@ichthyostega.de> wrote: > >>> Am 20.09.21 um 02:40 schrieb Ichthyostega: >>>> You can see my progress in the new Git repository published on Github >>>> https://github.com/Ichthyostega/yoshimi-test.git >> >> Am 08.10.21 um 02:23 schrieb Ichthyostega: >>> .... The basic idea is to use a simple linear regression to fit with the >>> basic samples vs runtime dependency over the whole test suite. This gives us >>> a prediction for each test, and relative to this prediction I store an >>> "expense factor" for each individual test case, so that >> >>> prediction * expense = expectedRuntime >> >> >> >> Hi Will and Kristian, >> >> now the question is, how *portable* are our Testsuite results... >> >> This afternoon, I systematically repeated the testsuite execution >> on a Laptop, which significantly differs in performance to my Desktop PC. >> >> The good news is: the concept works as intended! >> * you check out the testsuite, including the "Baseline data", >> which for Timings is the *Expense Factor* for each test case. >> * you run the suite a number of times to get an average >> * then you run once with "--calilbrate". >> This creates a new linear regression fit for the *platform model* >> * and after that, all Tests run GREEN, using the checked-in Expense Factors >> >> >> However, some Details are notable. >> The measurements are really different, and also slightly different in >> structure. >> The "GREEN" result can really only be achieved with the help of Statistics. >> >> The error margin for such measurements is quite high, >> after running several times, we typically see fluctuations around 10-20% >> >> This is no surprise, but common for measurements under real-world conditions. >> This well known fact is also the reason why we typically do not engage into >> micro-optimisations just to gain some % -- unless we're really desperate. >> >> >> See attached the corresponding graph, with similar legend than the image >> I showed you yesterday. Note the following >> >> - different Y axis, since everything is slower on that laptop >> - similar curve shapes, with the notable difference that the AddSynth >> becomes faster on that Laptop for the long note time (10s, right) >> >> - the "platform model" i.e. the (blue) regression line differs significantly >> >> Y = 6.61ms + 151ns/smp * samples >> >> (vs. my Desktop: Y = 4.38ms + 103ns/smp * samples) >> >> >> >> Another irritating observation however is, that I get a *sound difference* >> on the calculated WAV file. The difference is extremely small, >> peak Δ -121.511dB(RMS). I can not hear it, it becomes only apparent when >> applying an extreme amplification on the residual WAV. It is basically >> slightly coloured white noise, with an faint trace of the actual sound. >> >> However, there should not be *any* difference in the generated sound. >> But I am not sure, if some differences in the default values might have >> caused that difference, like a slightly different volume setting somewhere. >> >> >> >> -- Hermann > > As there are quite a lot of int(char?) to float calculations even running the > default sound, is it possible that these conversions are done slightly > differently across the different processors?
https://en.wikipedia.org/wiki/IEEE_754#Reproducibility seems to indicate that some variations are allowed, especially in the stored binary form. Given how small the difference is, I wonder whether this is on the brink of floating point precision, and that *numerically* they are still "essentially" zero. One thing we could do to remedy that is simply to convert it back to signed 16-bit integer format. This isn't a good format for processing, but it's good for a final render, and should get rid of these extremely tiny and insignificant differences. >> Which brings me to the next topic I have to address for that Testing >> endeavour: >> The Testsuite must launch Yoshimi with totally controlled default settings. >> We all know, the topic of configuration is complicated (sigh) ;-) Indeed. How hard would it be to have an option which sets the config directory? Aka "--config-dir=testsuite/config", and then there would be a "yoshimi-0.instance" file there. -- Kristian _______________________________________________ Yoshimi-devel mailing list Yoshimi-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/yoshimi-devel
