Jacques Menu <[email protected]> writes: > Hello folks, > > Sorry, this message is rather long… > > > I’ve been wondering what design decisions would be made, should > someone start a project like LilyPond today, with the years-long > experience we have behind us in this field. > > LilyPond can be characterized the following way, please excuse any > important omission or error: > - its native syntax is TeX inspired;
Wrong. Its earliest implementations used TeX as a backend and continued where MPP, a preprocessor for a TeX-based music system let off. As a consequence, early accent input in lyrics was TeX-based (it isn't anymore), there is % as a line comment character and \ for introducing a control sequence/word. That's about it. Very few lexical similarities that actually remain these days. You cannot change syntax on the fly (as with TeX), the input is structured and nested and has a grammar. > - the ground code is written in C++, with lexical and > syntactical analysis done with flex and bison, respectively; > - the extension language is Scheme, from the Lisp family of languages; > - the implementation is one-pass, meaning that the work done > when handling a file goes straight from the characters level to > graphics and MIDI output, without any intermediate stop; That's complete nonsense. I mean, just take a look at the console while LilyPond processes input: the input processing takes the least amount of time. Input is processed into data structures (score/book/markup), those are processed using hooks that generally store those data structures elsewhere, and at the end of that processing the structures actually get processed using "iteration" that runs through the temporal sequence of music expressions in a score and uses engravers for generating grobs and performers for generating MIDI elements, and at the end generating a PostScript and/or Midi file from them, with the PostScript then getting processed into other formats. Instead of PostScript, SVG might get written. To call this complex system with different passes and phases "one-pass" does not make remote sense. > - building LilyPond on various operating systems relies on GUB and > Python. Creating our official installers relies on that. Many LilyPond implementations (like those typically used in GNU/Linux systems) are compiled natively. > Some aspects of the current state of the Lily implementation are a bit > saddening: > - Scheme is not easy to read nor write for most users, who can > fortunately get help from the very useful and active users group; It's easy to read and write for the computer. It's a functional language that, due to the Guile implementation, interfaces very nicely with LilyPond. > - the extensive use of Scheme code in the implementation is > very flexible and powerful, but implies performance limitations; By far most time is spent in C++ code. The data structures might be somewhat of a runtime cost but they make the user-accessible programming model rather straightforward. > - the developpers and maintaince people we are lucky to have > have had much work recently with GUB and the move from Python 2.x to > 3.y; > - solving the famous #34 issue is nearly impossible in the > current one-pass setup. Again, this is complete nonsense since the not actually existent "one-pass setup" has nothing to do with the problem. The required behavior during iteration (which is the second pass of the "one-pass setup") is complicated by the semantics of grace notes as backward-in-time infinitesimal notes are hard to integrate consistently with time moving forward and causing contexts to be created. > There have been very interesting discussions about all this at the > Salzburg conference last January, which lead me to dream of things > being done another way in an ideal world: > - keep the TeX inspired syntax, of course; > - switch to another, easier to read and write extension > language. Someone suggested Lua, for example, which would be a good > candidate it seems; It isn't functional, so calling it one-off for expressions won't be as straightforward as Guile is. It doesn't have native data structures to represent syntax trees like Lisp dialects do: its principal mapping of programs and data is to plain text rather than nested lists, and that means that the _structure_ Lisp provides for the manipulation of programmatic concepts has to be designed and translated manually and that makes something like the seamless implementation of lexical closures across both LilyPond and Guile code close to impossible. It doesn't do arbitrary precision math by default (granted, LilyPond has a lot of C++ interfacing that doesn't either) which means that overflows will have weird effects and go without getting flagged. > - rely more on C++ in the implantation for speed, keeping the > extension language mainly for users in their files; This is already the case: almost everything that is of significant performance impact is done in C++. > - go for a multi-pass implementation, in which an internal > representation of the score is built in memory in a first move. Then > other moves produce graphics and MIDI output from this representation, > which could also be used to convert LilyPond to MusicXML or other > formats; Since that is already what happens, I have no idea what you think you are proposing. > - rely only on C++, flex, bison and the extension > language. Eliminating the need for GUB and Python would greatly > simplify porting LilyPond to various environments, including web > sites, phones and tablets. That's hopelessly naive. GUB exists exactly because porting to things like MacOSX and Windows natively is a lot of effort for an application relying on a typical Unix-style tool chain. Take a look at the manpower the Git project absorbs on a continuing basis for providing a somewhat current Windows port in parallel with its other releases. For LilyPond, Windows ports just fall out of the release process automagically. Until they don't, of course. But getting that fixed has consumed far less time and effort than having to maintain a complete current native port to Windows would have. > Multi-pass would help solve the #34 issue: a representation of the > whole score in memory can be used to spot the places where one has to > add grace skips as of today, before executing the graphics creation > pass(es). This is the way xml2ly adds such skips automatically to > avoid the issue in the LilyPond code it generates. Again, your proposal is completely disconnected from how LilyPond actually is already operating. > I’m ready to participate, should other people aim at such an ideal world. > > > Now to my question. > > We often have to add things such as short skips and cadenzas to avoid > having so-called events at the exact same time moment in the score. Who is "we"? It's comparatively infrequent in my experience, certainly not "often". > This is illustrated by \HiddenMeasureAndBarLine in the attached file: > commenting its use out at line 47 shows the problem. > > Would it be meaningful, useful and not too hard to implement, to allow > for several events at a given time moment, and a way to sort them > depending on the user's needs? This would make it possible for the > user to control the order in which LilyPond would handle events > occurring at this time moment. Time is sequential. If you are talking about typesetting order, there are various properties that influence stacking orders and spacing. > If so, such a feature could be added to the above wish list… Half of the wish list makes little sense in relation with what LilyPond currently does. Of course that may mean that what LilyPond currently does is not transparent and accessible in sufficient degree to a typical user. Changing that is certainly a worthwhile and partly an ongoing project. I've actually invested a whole lot of work to make the programmatic side of LilyPond match the user side of LilyPond better and more naturally and get rid of strange things concerning the mapping of music expressions to Scheme expressions. It may be worth looking at how TeX got anywhere. Every single project trying to reimplement TeX/LaTeX on top of some other programming system or code base has imploded from a lack of traction and too few participants keeping engaged (one of the more promising such attempts may have been ANT <https://www.ctan.org/pkg/ant>, an implementation of something inspired by LaTeX in OCaml). Ultimately what has actually made the race is LaTeX, a format built on top of the ancient code base. And in the wake of Context, a different format with a rather smaller basis of interested users and a different focus, parts of TeX have been refactored into something accessible with Lua. The integration of the Lua and TeX languages is not really a thing: Lua is not as much married to TeX in LuaTeX as it is its personal brain surgeon. And make no mistake: TeX/LaTeX have a much more widespread user base than LilyPond has. So in terms of expanding the user base and its comfort level, looking at high-level formats like LaTeX and formalising interfaces to make it possible to build applications and extensions independently (working with LilyPond is more akin to working with plain TeX rather than LaTeX) seems like a more promising course. -- David Kastrup
