On Fri, 8 Oct 2021 19:44:48 +0200 Ichthyostega <p...@ichthyostega.de> wrote:
>> Am 20.09.21 um 02:40 schrieb Ichthyostega: >>> You can see my progress in the new Git repository published on Github >>> https://github.com/Ichthyostega/yoshimi-test.git > >Am 08.10.21 um 02:23 schrieb Ichthyostega: >> .... The basic idea is to use a simple linear regression to fit with the >> basic samples vs runtime dependency over the whole test suite. This gives us >> a prediction for each test, and relative to this prediction I store an >> "expense factor" for each individual test case, so that > >> prediction * expense = expectedRuntime > > > >Hi Will and Kristian, > >now the question is, how *portable* are our Testsuite results... > >This afternoon, I systematically repeated the testsuite execution >on a Laptop, which significantly differs in performance to my Desktop PC. > >The good news is: the concept works as intended! > * you check out the testsuite, including the "Baseline data", > which for Timings is the *Expense Factor* for each test case. > * you run the suite a number of times to get an average > * then you run once with "--calilbrate". > This creates a new linear regression fit for the *platform model* > * and after that, all Tests run GREEN, using the checked-in Expense Factors > > >However, some Details are notable. >The measurements are really different, and also slightly different in >structure. >The "GREEN" result can really only be achieved with the help of Statistics. > >The error margin for such measurements is quite high, >after running several times, we typically see fluctuations around 10-20% > >This is no surprise, but common for measurements under real-world conditions. >This well known fact is also the reason why we typically do not engage into >micro-optimisations just to gain some % -- unless we're really desperate. > > >See attached the corresponding graph, with similar legend than the image >I showed you yesterday. Note the following > >- different Y axis, since everything is slower on that laptop >- similar curve shapes, with the notable difference that the AddSynth > becomes faster on that Laptop for the long note time (10s, right) > >- the "platform model" i.e. the (blue) regression line differs significantly > > Y = 6.61ms + 151ns/smp * samples > > (vs. my Desktop: Y = 4.38ms + 103ns/smp * samples) > > > >Another irritating observation however is, that I get a *sound difference* >on the calculated WAV file. The difference is extremely small, >peak Δ -121.511dB(RMS). I can not hear it, it becomes only apparent when >applying an extreme amplification on the residual WAV. It is basically >slightly coloured white noise, with an faint trace of the actual sound. > >However, there should not be *any* difference in the generated sound. >But I am not sure, if some differences in the default values might have >caused that difference, like a slightly different volume setting somewhere. > > >Which brings me to the next topic I have to address for that Testing endeavour: >The Testsuite must launch Yoshimi with totally controlled default settings. >We all know, the topic of configuration is complicated (sigh) ;-) > >-- Hermann As there are quite a lot of int(char?) to float calculations even running the default sound, is it possible that these conversions are done slightly differently across the different processors? The idea has been mooted before of changing all the 'value' parameters to floats, so the only time we'd need conversions would be for incoming MIDI and for loading legacy instrument files (we already have a way to reliably save floats in XML files). Would that eliminate such errors? At least, it's encouraging that the sound differences are so small - and nice to know we have a way to test it :) Will. -- Will J Godfrey https://willgodfrey.bandcamp.com/ http://yoshimi.github.io Say you have a poem and I have a tune. Exchange them and we can both have a poem, a tune, and a song. _______________________________________________ Yoshimi-devel mailing list Yoshimi-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/yoshimi-devel
