On January 22, 2014 02:45:08 AM Florian Jung wrote: > Hi all, > > Lots of things in MusE are pretty complicated, because they involve > communication with the audio thread, or because they require us to > pre-allocate things outside of the audio thread and so on. > > I want to question this realtime capability of MusE, please bear with me: > > > My first point to legitimate all this is: > > MusE aims to be realtime capable, but *it is not*. > > What makes me say that? Well, even though realtime patterns are > implemented in MusE (such as message passing with the audio thread, > because we can't use mutexes, and pre-allocating buffers outside), they > are not consequently used. Some examples: > > when undo()ing, this can take an undetermined amount of time inside the > audio thread; it basically scales in O( number_of_operations_to_undo ), > which might be large (e.g. if i have deleted 1000 events at once.). MusE > might fail to meet the realtime constraints -> glitch. > > in many places, MusE does tempomap lookups *in the audio thread*. C++'s > map structure, however, does not make any guarantees about it's > worst-case-runtime. If we're unlucky, we might fail to meet our realtime > constraints. (I must admit, a large part of this is my fault.) > > Every frame, MusE collects the events to play in the next buffer period > by finding begin and end iterator in the EventList, and loops through > lots of data structures. Most of these are from the STL and do not make > any timing guarantees. Additionally, it's plain unnecessary. One could > just increment an iterator (which, of course, still needs to be an > iterator in a realtime-safe structure), and a search-in-binary-tree > would only be necessary upon a seek(). > > The Audio thread may not allocate any memory, therefore, lots of > uglyness is done to do this allocation outside of the audio thread (e.g. > by the gui thread, and then only passing (smart) pointers). > However, then, newly created events are inserted into a std::multimap. > Which internally needs to malloc() a new node of the binary tree (or > whatever internal data representation is used). Malloc is known to not > fulfill any realtime constraints -> muse might glitch. > > Certain things are done by switching the audio thread into Idle state, > which stops processing any data, and instead delivers zeros. > While MusE will not fail at realtime here, this still introduces > glitches. We may not do this, msgIdle must be eliminated. > > > > This puts me in a dilemma: > > a) we need to *massively* redesign MusE to make it realtime capable. > This would involve hard separation between audio and GUI, no shared > data! They should be put into different processes, even. > It would probably affect GUI performance. > We must restrict editing operations while playing back. Forbid tempo > changes, forbid "large" event manipulations while playing. > b) We drop this hard realtime thing. MusE will become a studio software > which should not, but *might* glitch. We would greatly reduce code > complexity by not having to respect all the corner cases. We will > keep the message passing, but we will use certain realtime-breaking > functions, like std::map's functions. > > I can't really see another way out of this. > > Actually, i like a), but i fear that this will be too much work; i think > that would almost equal a full rewrite from scratch. > > Could we probably even use mutexes for synchronisation? I mean, our > critical blocks when editing are pretty short, so maybe priority > inversion could help us? > > With priority inversion, when a low-prio thread holds a mutex that is > required by a high-prio thread, then it temporarily also gains high > priority until the mutex is resolved; then control is given to the > high-prio thread. > > Please discuss, I'm totally insecure about this :/ > Florian
Relax, you are right. It has taken me a long time but I realize now that the way MusE handles processing is wrong. I mean I've known for some time that the heavy processing in the process callback was a bit weird. I watch LAD and know that others use threads to pump data in and out of the Jack process callback (I recall at least one app uses multiple threads, impulse response app I think). But I was not quite sure what to do, or why ours was so wrong. The last week or so, examining the code I'm thinking more about it now. To make a long story (which follows) short, we'll do it the right way, something like this: https://github.com/jackaudio/jack2/blob/master/example-clients/capture_client.c A bit of back-story may help to understand the current MusE situation: --------------- I coded Windows audio apps since early '90s. Both MMSystem and DirectSound. And I used mutexes (Critical Sections). I dug up my very old code yesterday. I started using Linux in 1997. I didn't code much of my own except fixing small bugs locally, until mid 2000s. I'm proud to have some fixes in the kernel - the SBLive driver and supporting DSP loader and GUI. Was very challenging. By around 2005 I had tried MusE, and the others. I said before probably one of the best reasons for choosing MusE was simply that input 'Transpose' box. Nobody else has it, even today I think. I could tell it was a good app, most resembling what I was used to - Cakewalk. I set about (not by choice!) fixing some painfully obvious crashes/bugs. But I was still a Linux audio app novice. Werner *already* had MusE-2 going (now muse-evolution in attic), but it was in heavy development but not really usable. Whereas MusE-1 was at least usable. And I needed something stable *NOW*, if ya know what I mean. So I chose MusE-1, as a sort of 'proving ground' - if I could fix bugs in MusE-1, learn how it works, maybe help Werner later, not step on his toes. But work on MusE-2 stalled, I think because he moved on to MusE-Score. My fixes made MusE-1 stable which is what we all needed - as musicians. If you think we do bad threading things now (you're right but) you should have seen all the many baaad crashes of threading things going on before. Like, things all over the place were manipulating STL containers while other threads were erasing items from them. Ugly. Do I care if I simply manipulate a boolean variable across threads? (Which I have done). I researched it recently and found, well, it's relatively safe but bad practice. We've never had a bug/crash because of it. IIUC some OSs like Linux and even CPUs atomize such fundamental operations. I tend to expect or hope that they would, not wanting to deal with it myself. What's your understanding of this, and is it a good thing? Shortly before you joined, Qt3 was dying very quickly, like within weeks. We talked about what to do. MusE-1 was Qt3 at the time. As MusE-2 was already a Qt4 app, I said "Well, I might as well abandon all work now and move over to Werner's MusE-2." But we all agreed that it was gonna be a /long/ time to make MusE-2 stable. In fact when I asked, the consensus was "Sick with the MusE-1 code base". Then something funny happened. Robert used Qt4's 'automatic' porting tool, just for a laugh, on MusE-1-Qt3. He said "Well I dunno if this is good, many things still broken...", shaking his head with sinking dismay, metaphorically speaking. Well, I took that hard work and /ran/ with it. And just then, Orcan joined and we all ported the thing over to Qt4. I guess what I'm saying is don't be too critical of the path chosen... So here we are. Stable. Usable. Correct? Expandable? Cutting edge? Mm... ----------------- Moving on... I believe we can make our process engine proper in... a few weeks, after discussing to make sure we know what we're doing and it'll work. Rough blueprint: ============== The Jack process callback should do one job only: Move data into and out of the Jack ports. You know all that crap we do in our process callback? That's gotta be moved outta there into OUR audio thread. As the Jack capture_client example shows above, one should really only use ring buffers (FIFO queues), and such, to PUMP data from/to a 'worker' thread (our audio thread) to/from the Jack ports, during the Jack process callback. And that's about all. Not all the stuff we do. I'm now seriously wondering why we have an audio thread at all, if all our heavy processing is done in our Jack process callback. I think, at one time, MusE was headed in the right direction but took a wrong turn somewhere. Someone long ago had actually linked the audio thread with Jack's thread using a pthread join or something like that. Sletz from the Jack team saw that one day, when I asked him a question. He gave us such *shit*. I never heard him *swear* loudly and publicly before or since. With these new changes, now the audio thread really MATTERS: Basically we keep our Audio::process() method and modify it, but now call it only from within our audio thread, gathering data and so on, putting/pulling data from the ring buffers. One thing I thought of: Once you put data to a ring buffer you can't take it out again from the same thread. So during a transport seek, our Jack process callback would need an indication of this seek condition to clear and ignore any 'stale' older invalid ring buffer items. Jack has such functions to check transport state. Now, there's the question of synchronization. Some operations in the audio thread may need to wait until our Jack callback has finished or has data, such as 'live' input effects operations that need input data first. Yet some operations need only just gingerly pre-pump items into the ring buffers without waiting - perhaps playing a wave file, as a rough example. We should consider what exactly we need the audio thread to do for us... Here I will interject that if we were to keep the current system, I was going to install a ring buffer for all 'operations' (in the undo/redo system) which only require Song::executeOperationGroup2 and Song::revertOperationGroup2. That is, operations like UndoOp::AddEvent or UndoOp::ModifyEvent do not require stage 1 or stage 3 non-realtime operations. It is really stupid that many simple operations wait for the audio thread. Rule of thumb, as you say Flo: The GUI should be de-coupled from audio so that almost NOTHING in the GUI needs to wait for ANYTHING. Careful timing, queuing, signaling, and stream inhibiting upon mouse presses, will help. For example: When somebody turns a midi controller knob in the controller panels, Audio::msgPlayMidiEvent() is called which waits at /each/ movement. Stupid baaad slooow, with large Jack periods. Using a ring buffer, the knob is free to turn at full speed and pump changes into the ring buffer and whatever reads the ring buffer deals with the changes, possibly even at a much slower rate than the knob is turning. See how I did this with all the /audio/ controls such as volume and pan. I mean crap, we even have an audio-thread-to-GUI-thread signaling mechanism (see Song::seqSignal()) which theoretically could be used to trigger the Song::update() calls that would be sooo needed, after a ring buffer 'operations' item handling would be finished in the process callback. (But it's a poor example that uses a pipe with single character messages.) Anyways we won't try to pursue that route now... If you want to start a new branch I'll try to work out some details and let's talk about how to do this. I think it would be relatively straight forward... Here's to "Doin' it right on the wrong side of town" (old Canadian hit song). --- I leave you with an idea to chew on: While looking at the undo/redo code I realized it can be destructive - changes can be lost if you undo, then change something, redo turns off. I thought "What if we could keep everything, discard nothing?" Thus I give you... Undo "Trees". Turns out emacs and vim have had "Undo Trees" for years. But apparently not many other apps. Look it up! As you can imagine, it needs a viewer of some kind, which emacs/vim have. Imagine being able to go to and from ANY modified state from ANY time in the project's entire history ! --- Thanks. Tim. ------------------------------------------------------------------------------ CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments & Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk _______________________________________________ Lmuse-developer mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/lmuse-developer
