On Wed, 26 May 2021 18:49:07 +0200
Ichthyostega <p...@ichthyostega.de> wrote:
>>> On 25.05.2021 17:07, Ichthyostega wrote:
>>>> Currently I'd prefer to build some dedicated test commands into the CLI.
>>>> But obviously, having a dedicated commandline argument to launch a test
>>>> would be another option. A third (less preferrable IMHO) option would be
>>>> to build a dedicated test executable, which links against yoshimi code.
>
>> On Wed, 26 May 2021 10:53:14 +0200 Kristian Amlie <krist...@amlie.name>
>> wrote:
>
>>> What about using LV2? I believe it checks off most of the boxes your
>>> brought up
>...
>...
>>> * Somewhat secondary but still a bonus: We test LV2 in the process.
>>>
>>> * Downside: If you don't know LV2, it is kind of hard to understand at
>>> first, but it does make sense!
>
>
>Am 26.05.21 um 12:05 schrieb Will Godfrey:
>> Another way, you could immediately get very complete access, is running
>> scripts from the CLI. These are plain text files such as the one below. It
>> has a deliberate error to demonstrate how it behaves. It occurs to me that we
>> could easily add a new command 'seed {n}' which would set random to a known
>> value for just this sort of thing.
>
>
>
>Hi Kristian,
>Hi Will,
>
>very helpful observations; using LV2 seems like another very interesting angle.
>In a similar vein, I recall a talk from LAC Berlin by Fons Adrianssen, where
>he demonstrated his framework for audio testing based on Jack + Python scripts
>https://lac.linuxaudio.org/2018/pages/event/46/
>
>
>After some more time spent thinking over those matters...
>Probably I should expand on my preferences for the testing approach -- which
>are based on my own experience plus a body of common knowledge about testing.
>Because, when a test setup is brittle, the maintenance can become a real
>burden.
>And the pattern of the "test pyramid" helps to mitigate those tensions.
>
>Basically we have two conflicting goals:
>- as much as possible, we want to run "the real thing" in the tests
>- but, the simpler the scaffolding, the higher the chances it can evolve
> alongside with the code and thus stay alive.
>
>And the typical solution ("test pyramid") means to *decompose* the
>test-subject.
>That is, we try to cover as much ground as possible with smaller units of
>functionality, which are easier and more robust to test. While the more
>complicated integration tests, which are hard to setup and to maintain,
>only cover the proper integration of those smaller units.
>
>To translate that into the situation with yoshimi: we should strive at covering
>as much ground as possible through direct calls into SynthEngine, noteOn/Off
>and
>to SynthEngine::MasterAudio(). Put another way, I think it is even /desirable/
>to take the Data-IO and MIDI processing out of the equation for the majority of
>the functionality tests. Because the interplay of CPU bound and IO bound
>processing can be complex and highly dependent on the setup and the hardware.
>
>And here I agree with Will: the most natural way to hook that in would be
>through the CLI. The test suite would thus use some scripting to organise
>the presets and baseline samples and then launch a given yoshimi release
>executable with a CLI script. Have still to figure out the details, but
>it seems we can pipe in the "run /path/to/script' into STDIN of yoshimi
>and the rest should then work automatically.
>
>My intention thus is to build some new CLI functions into standard yoshimi
>- set seed (I really like that idea!)
>- calculate one test note and pipe the results out into a file descriptor
>- calculate N test notes and just throw the audio away but capture the time.
>
>
>Besides that, we really also should try to complement that approach by builing
>an integration testing setup. Kristian's idea really looks promising here: we
>could build a very rudimentary LV2 host (from existing sample code), launch
>Yoshimi as a plugin and produce some simple and totally predictable wave forms,
>probably with timing measurements.
>
>-- Hermann
LV2 would indeed give sample accurate control which could help with
repeatability using a dedicate host implementation. As Kristian said, it would
at the same time test our LV2 code. However, we don't currently make much
direct control available, just MIDI I think.
MIDI events pass through the same system as everything else, and when the CLI
still had direct numeric entries, I was using it to test note on/off behaviour
as well as CCs.
--
Will J Godfrey
https://willgodfrey.bandcamp.com/
http://yoshimi.github.io
Say you have a poem and I have a tune.
Exchange them and we can both have a poem, a tune, and a song.
_______________________________________________
Yoshimi-devel mailing list
Yoshimi-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/yoshimi-devel
