Re: Differences in `web.html`
Jonas wrote: >> I see the attached difference. Is it an oversight that >> `scripts/build/fix-docsize.sh` is not executed while building the >> website on 'lilypond.org'? > > Yes, intentionally so: "make website" does not execute it because it > doesn't have access to the linked manuals and cannot fill in its > size. Based on the path above, you've been executing "make doc" > which also generates the website as part of it. Btw, there are a > number of other differences, search for "web_version" in the > Documentation/web/* files. Han-Wen wrote: > note that the website isn't built on lilypond.org anymore. Rather, > there is a cronjob that downloads an artifact from gitlab, see > https://gitlab.com/hahnjo/lilypond-infrastructure/-/blob/master/website/main.go Thanks for the explanations. Do I understand correctly that we have a nifty feature (namely showing the file size to download) that stays unused because it would be too expensive for deployment, both CPU- and time-wise? Werner
Re: Anybody else with an interest in parser wrangling?
On 3/20/23 15:35, David Kastrup wrote: Well, we are talking about core maintenance and rearchitecting here. The main objective in my book is making more people able to solve their own problems with confidence without ever having to touch the core. That involves making sure that "the way things are done" is straightforward and coherent and does not require to interact with any non-obvious bits and pieces. Thank you. Please help me understand your boundary between core and non-core. In you first message you specifically mentioned parser.yy. Is the lexer/parser process your focus? The taste and tenacity to leave something cleaner behind than what they started with. But that sounds like the qualifications for a single person, and there might be incentive for some small team to work on stuff with distributed responsibilities. In a very real sense, your qualifications describe many of the people who populate this mailing list. Given the history of lily/parser.yy you shared earlier, developing a cohort of people (small team) with interests similar to yours seems attractive for LilyPond. As an initial step toward such a team, is there anything in particular David Zelinsky or I could do to help your current efforts? Perhaps, if you have a some specific improvements you think are worth pursuing we could see how well we could contribute. Thank you, John Wheeler
PATCHES - Countdown to March 22
Here is the current countdown report. The next countdown will begin on March 22nd. A list of all merge requests can be found here: https://gitlab.com/lilypond/lilypond/-/merge_requests?sort=label_priority Push: No patches in Push at this time. Countdown: !1877 Spelling issues - Werner Lemberg https://gitlab.com/lilypond/lilypond/-/merge_requests/1877 !1875 Axis_group_interface::combine_skylines: Add missing translation - Jean Abou Samra https://gitlab.com/lilypond/lilypond/-/merge_requests/1875 !1874 GNUmakefiles: replace bash dependent code - John Wheeler https://gitlab.com/lilypond/lilypond/-/merge_requests/1874 !1872 Volta_engraver cleaning - Dan Eble https://gitlab.com/lilypond/lilypond/-/merge_requests/1872 !1871 Fix segfault in Custos_engraver - David Kastrup https://gitlab.com/lilypond/lilypond/-/merge_requests/1871 !1867 \qr-code markup command - Jean Abou Samra https://gitlab.com/lilypond/lilypond/-/merge_requests/1867 Review: !1876 musicxml2ly performance improvements - Dan Eble https://gitlab.com/lilypond/lilypond/-/merge_requests/1876 !1873 Enhance white mensural ligatures - Benkő Pál https://gitlab.com/lilypond/lilypond/-/merge_requests/1873 New: No patches in New at this time. Waiting: !1810 Let \rhythm markup derive from current layout - David Kastrup https://gitlab.com/lilypond/lilypond/-/merge_requests/1810 yr humble & obd't, Colin
Re: Is it time to update the Finale 2008 sample in Essay?
Han-Wen Nienhuys writes: > On Mon, Mar 20, 2023 at 5:27 PM Abraham Lee > wrote: > >> Thanks, Jean, for doing that. I was hoping for a more public discussion to >> see if creating an issue is even warranted. The essay is a historical >> document, to be sure, so updating the comparison files might not be needed >> at all. > > I agree to this. It is better to simply put a small intro to the essay > that gives the context. > > We could then add an updated preface/postscriptum that puts it in > context for 2023. To note, a lot of modern music engravers have taken > inspiration from the LilyPond attitude to engraving. The Dorico blog > posts have been quite explicit about it, and maybe we could ask the > MuseScore folks for comments too. For better or worse, I think the main selling point of LilyPond these days is not as much quality as workflow. -- David Kastrup
Re: Is it time to update the Finale 2008 sample in Essay?
On Mon, Mar 20, 2023 at 3:25 PM Jean Abou Samra wrote: > Le lundi 20 mars 2023 à 10:26 -0600, Abraham Lee a écrit : > > Thanks, Jean, for doing that. I was hoping for a more public discussion to > see if creating an issue is even warranted. The essay is a historical > document, to be sure, so updating the comparison files might not be needed > at all. It just feels a bit odd to read "we have chosen Finale 2008, which > is one of the most popular commercial score writers". This was absolutely > true... once upon a time. Reading it now makes it sound like we had to dig > way back in order to pretend to make it seem like Finale isn't good > enough and that LilyPond does it right. How does > Finale/Sibelius/Dorico/etc. do nowadays? Do they get it right now? I'm > certain folks have asked this question. > > For comparison, I just entered the two systems in the essay into MuseScore > 4 and got a practically perfect output. Entering one voice at a time (voice > 1, then voice 2), all existing pitches were maintained in voice 1 despite > making alterations in voice 2 (like that omitted flat that Finale 2008 > leaves out). I didn't have to correct or add anything that was missing. > Maybe I just got lucky because of how I entered the passage. Arguments can > be made about other layout decisions, but I think it's hard to argue > against what MS4 has done compared to the hand-engraved examples: > > So, maybe all that's needed is a different wording in this section to > reflect why *at the time* this comparison made sense (like what is > described at the beginning of the essay)? That would certainly be simpler > than recreating the comparison (which might not come to the original > conclusion like it used to). > > LilyPond too has evolved a lot in 15 years. You could take a more complex > example than this relatively simple (in terms of notation) Bach excerpt, > and re-do the comparison. I'm not sure Finale/Sibelius/Dorico/MuseScore > would use skylines for spacing objects as opposed to simple boxes. > It most certainly has, in so many excellent ways. I've used all of these major apps to some degree over the past number of years and I have discovered there are many things about how each app lays out a page that really frustrate me, some completely hiding the control to force things onto a specific page/system/etc. This is one big reason I continue to use LP after all these years. The layout control is simply superb! My only complaint here is that there isn't a great mechanism to finely control the system placement on a page (or staves/lyrics/etc. within a system) aside from explicit vertical placement, which I avoid completely. I wish there was a more convenient way to do a similar thing like \once \override TextScript.X-offset = #5 to shift a grob and then have things re-flow around it (as opposed to what extra-offset does). Sorry, off on a tangent there. I read that a while ago, Urs Liska organized a "music engraving contest" > where scores were compared in different score writers, probably for the > scores of beauty blog. That blog is now defunct, but you could try to dig > into the list archives. It's old, but not 15 years old, and the chosen > samples could be recompared today. > Yes, those were good times lol. I actively participated in those, partially from the sidelines, but some on the front lines. Those contests were difficult to run in a controlled way. Meaning, what is actually being compared? How do you know who wins? How "out of the box" are we comparing other software to LP? With LP, it's easy, just don't use any overrides and you get default behavior. In other apps, the way things show up are very dependent on how the entry takes place. And an expert user of software X can do just as well a job as an expert user of software Y. So, what actually is being compared in the end? Time to get to a specific result? One user's ability to use their favorite software against another's? This seemed to be the challenge we ran into because these were always the questions that came up. Complaints one way or the other were always about nuanced differences in output or expectations rather than gross errors by the application despite a user's best effort. I'm not against this, but I'm not sure how to make this a practically useful activity, either.
Re: Is it time to update the Finale 2008 sample in Essay?
Le lundi 20 mars 2023 à 10:26 -0600, Abraham Lee a écrit : > Thanks, Jean, for doing that. I was hoping for a more public discussion to > see if creating an issue is even warranted. The essay is a historical > document, to be sure, so updating the comparison files might not be needed at > all. It just feels a bit odd to read "we have chosen Finale 2008, which is > one of the most popular commercial score writers". This was absolutely > true... once upon a time. Reading it now makes it sound like we had to dig > way back in order to pretend to make it seem like Finale isn't good > enough and that LilyPond does it right. How does Finale/Sibelius/Dorico/etc. > do nowadays? Do they get it right now? I'm certain folks have asked this > question. > > For comparison, I just entered the two systems in the essay into MuseScore 4 > and got a practically perfect output. Entering one voice at a time (voice 1, > then voice 2), all existing pitches were maintained in voice 1 despite making > alterations in voice 2 (like that omitted flat that Finale 2008 leaves out). > I didn't have to correct or add anything that was missing. Maybe I just got > lucky because of how I entered the passage. Arguments can be made about other > layout decisions, but I think it's hard to argue against what MS4 has done > compared to the hand-engraved examples: > > > > So, maybe all that's needed is a different wording in this section to reflect > why *at the time* this comparison made sense (like what is described at the > beginning of the essay)? That would certainly be simpler than recreating the > comparison (which might not come to the original conclusion like it used to). LilyPond too has evolved a lot in 15 years. You could take a more complex example than this relatively simple (in terms of notation) Bach excerpt, and re-do the comparison. I'm not sure Finale/Sibelius/Dorico/MuseScore would use skylines for spacing objects as opposed to simple boxes. I read that a while ago, Urs Liska organized a "music engraving contest" where scores were compared in different score writers, probably for the scores of beauty blog. That blog is now defunct, but you could try to dig into the list archives. It's old, but not 15 years old, and the chosen samples could be recompared today. signature.asc Description: This is a digitally signed message part
Re: Musings on our translated documentation build
On Sun, 2023-03-19 at 23:56 +0100, Federico Bruni wrote: > Actually I was curious to check the html output rather than the changes > in the source. > If it doesn't break working links, it's fine for me. In the parts I touched, it is almost generating identical HTML (except for the links to unnumbered sections; but I'm coming to the conclusion that probably want to stop treating them specially anyway). I will follow up on this once I have more time. > If you eventually proceed, write the scripts, etc. I'd be happy to > check if the Italian manuals look good and links work correctly. CI > builds the doc also, right? Yes, the documentation gets built for every merge request, but it's not available for inspection (and also pretty big). > Maybe English manuals only? Not sure I understand this question, CI builds all languages. English is also pretty boring with respect to this change because it's not translated and doesn't have @translationof and the related machinery. Jonas signature.asc Description: This is a digitally signed message part
Re: Is it time to update the Finale 2008 sample in Essay?
On Mon, Mar 20, 2023 at 3:13 PM Han-Wen Nienhuys wrote: > On Mon, Mar 20, 2023 at 5:27 PM Abraham Lee > wrote: > > > > On Sat, Mar 18, 2023 at 6:17 AM Jean Abou Samra > wrote: > > > > > Le mercredi 15 mars 2023 à 00:57 +0100, Jean Abou Samra a écrit : > > > > > > Le mardi 14 mars 2023 à 17:44 -0600, Abraham Lee a écrit : > > > > > > At the time I started with LP, the Finale 2008 example wasn't that old > > > in the Essay section. Is it really fair for us to continue showing this > > > since quite a few versions have been released since then? I mean, > Finale 27 > > > has been out since June 2021 and is now at 27.3. I'm not saying the > > > example is bad, nor doesn't it illustrate a historical > > > piece of evidence clarifying why LP was needed. I guess I'm wondering > if > > > it's worth creating a new set of examples to show why it's *still* > needed, > > > even after all these years? Thoughts? > > > > > > I suggest you open a tracker issue. > > > > > > I have done it now, https://gitlab.com/lilypond/lilypond/-/issues/6547 > > > > > > > Thanks, Jean, for doing that. I was hoping for a more public discussion > to > > see if creating an issue is even warranted. The essay is a historical > > document, to be sure, so updating the comparison files might not be > needed > > at all. > > I agree to this. It is better to simply put a small intro to the essay > that gives the context. > > We could then add an updated preface/postscriptum that puts it in > context for 2023. To note, a lot of modern music engravers have taken > inspiration from the LilyPond attitude to engraving. The Dorico blog > posts have been quite explicit about it, and maybe we could ask the > MuseScore folks for comments too. > Hey, Han-wen! Thanks for chiming in! You would certainly know better than anyone the context of the statements/examples in the essay. And, yes, the other apps certainly have taken inspiration from LP's output, which is part of why I wasn't sure which the right answer would be. I am strongly leaning towards modifying the context of what's there to keep things in the appropriate perspective with the competition. Best, Abraham
Re: Differences in `web.html`
On Mon, Mar 20, 2023 at 3:35 PM Werner LEMBERG wrote: > > > Comparing > > http://lilypond.org/web.html > > with a self-compiled > > .../out-www/offline-root/Documentation/web/web.html > > I see the attached difference. Is it an oversight that > `scripts/build/fix-docsize.sh` is not executed while building the > website on 'lilypond.org'? note that the website isn't built on lilypond.org anymore. Rather, there is a cronjob that downloads an artifact from gitlab, see https://gitlab.com/hahnjo/lilypond-infrastructure/-/blob/master/website/main.go -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Differences in `web.html`
On Mon, 2023-03-20 at 14:34 +, Werner LEMBERG wrote: > Comparing > > http://lilypond.org/web.html > > with a self-compiled > > .../out-www/offline-root/Documentation/web/web.html > > I see the attached difference. Is it an oversight that > `scripts/build/fix-docsize.sh` is not executed while building the > website on 'lilypond.org'? Yes, intentionally so: "make website" does not execute it because it doesn't have access to the linked manuals and cannot fill in its size. Based on the path above, you've been executing "make doc" which also generates the website as part of it. Btw, there are a number of other differences, search for "web_version" in the Documentation/web/* files. > PS: That we now see 'web.pdf' instead of 'Web.pdf' seems to be a > consequence of commit d638af5410936d255f. No, this has nothing to do with that - it's not even a button in a navigation bar! It is because of weblinks.itexi and which macros are used depending on "web_version" (see above). signature.asc Description: This is a digitally signed message part
Re: Is it time to update the Finale 2008 sample in Essay?
On Mon, Mar 20, 2023 at 5:27 PM Abraham Lee wrote: > > On Sat, Mar 18, 2023 at 6:17 AM Jean Abou Samra wrote: > > > Le mercredi 15 mars 2023 à 00:57 +0100, Jean Abou Samra a écrit : > > > > Le mardi 14 mars 2023 à 17:44 -0600, Abraham Lee a écrit : > > > > At the time I started with LP, the Finale 2008 example wasn't that old > > in the Essay section. Is it really fair for us to continue showing this > > since quite a few versions have been released since then? I mean, Finale 27 > > has been out since June 2021 and is now at 27.3. I'm not saying the > > example is bad, nor doesn't it illustrate a historical > > piece of evidence clarifying why LP was needed. I guess I'm wondering if > > it's worth creating a new set of examples to show why it's *still* needed, > > even after all these years? Thoughts? > > > > I suggest you open a tracker issue. > > > > I have done it now, https://gitlab.com/lilypond/lilypond/-/issues/6547 > > > > Thanks, Jean, for doing that. I was hoping for a more public discussion to > see if creating an issue is even warranted. The essay is a historical > document, to be sure, so updating the comparison files might not be needed > at all. I agree to this. It is better to simply put a small intro to the essay that gives the context. We could then add an updated preface/postscriptum that puts it in context for 2023. To note, a lot of modern music engravers have taken inspiration from the LilyPond attitude to engraving. The Dorico blog posts have been quite explicit about it, and maybe we could ask the MuseScore folks for comments too. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Anybody else with an interest in parser wrangling?
John Wheeler writes: > On 3/19/23 11:51, David Kastrup wrote: >> So how to better involve others? > > Maybe a good place to start is by asking a few questions. > > What you would like these others to do? Well, we are talking about core maintenance and rearchitecting here. The main objective in my book is making more people able to solve their own problems with confidence without ever having to touch the core. That involves making sure that "the way things are done" is straightforward and coherent and does not require to interact with any non-obvious bits and pieces. > What qualifications would they need? The taste and tenacity to leave something cleaner behind than what they started with. But that sounds like the qualifications for a single person, and there might be incentive for some small team to work on stuff with distributed responsibilities. -- David Kastrup
Fwd: Interested in GNU Lilypond Google Summer of Code Beaming Fixing Project Idea
Forwarding to list, since I apparently didn't add the list to the email as I intended to. -- Forwarded message - From: Carl Sorensen Date: Mon, Mar 20, 2023 at 12:59 PM Subject: Re: Interested in GNU Lilypond Google Summer of Code Beaming Fixing Project Idea To: Jason Yip Jason, Thanks for your interest! On Sun, Mar 19, 2023 at 12:21 AM Jason Yip wrote: > Hello Carl, > > > My name is Jason Yip, and I am interested in learning more about GNU > Lilypond's "Fix Beaming Patterns/Beam Subdivisions and Tuplets" project > idea for Google Summer of Code and participating in this project idea. I > would like to ask, are you are still looking for prospective 2023 GSoC > contributors for this project idea? > > I am currently a 2nd year student at University of Illinois at > Urbana-Champaign, US and I am studying Computer Science + Music. I have > some basic familiarity with LilyPond as I sometimes use it to create > personal scores. I have about 4 years of experience in C++, which I hope > will be of use. I am available to contribute full-time during the summer > and would like to explore this issue, even if it means refactoring a lot > of the Beam C++ classes. > The beaming project would be a great asset to LilyPond. I’d love to see you tackle it, if you’re interested. Your background in C++ would be a great asset here. In preparation for submitting a proposal, it might be good to rename things in beaming-pattern.cc There is a discussion on the lists (wow, from 5 ½ years ago!) that mentions the names used in the code don’t match the names used in the user documentation: https://lists.gnu.org/archive/html/lilypond-devel/2017-11/msg00037.html The bottom line on that discussion is that we could change “group” in beaming-pattern.cc to “beat”, and “beat” in beaming-pattern.cc to “ base_moment” to match our user documentation. I would expect that making this change would do two things: 1. Get you an introduction to beaming-pattern.cc, which is where the major work needs to be done, and 2. Get you an introduction to our process for handling merge requests, which would make you part of the team. I've added lilypond-devel@gnu.org on the reply, in order to get you introduced to the development group. I'd also suggest that you look at the LilyPond Contributor's Guide, which is where we keep all the available information about our development processes. If you'd like some help in getting started on the terminology change project, please let me know. Thanks, Carl I
Re: Anybody else with an interest in parser wrangling?
On 3/19/23 11:51, David Kastrup wrote: So how to better involve others? Maybe a good place to start is by asking a few questions. What you would like these others to do? What qualifications would they need? John Wheeler
Re: Interested in GNU Lilypond Google Summer of Code Beaming Fixing Project Idea
Jason, I neglected to leave a link to the Contributor's Guide: https://lilypond.org/doc/v2.25/Documentation/contributor/index.html Thanks, Carl On Sun, Mar 19, 2023 at 12:21 AM Jason Yip wrote: > Hello Carl, > > > My name is Jason Yip, and I am interested in learning more about GNU > Lilypond's "Fix Beaming Patterns/Beam Subdivisions and Tuplets" project > idea for Google Summer of Code and participating in this project idea. I > would like to ask, are you are still looking for prospective 2023 GSoC > contributors for this project idea? > > I am currently a 2nd year student at University of Illinois at > Urbana-Champaign, US and I am studying Computer Science + Music. I have > some basic familiarity with LilyPond as I sometimes use it to create > personal scores. I have about 4 years of experience in C++, which I hope > will be of use. I am available to contribute full-time during the summer > and would like to explore this issue, even if it means refactoring a lot > of the Beam C++ classes. > > I have also attached my resume. > > Thank you for considering my interest and I hope to hear back from you! > > -- > - Jason Yip > >
Re: Is it time to update the Finale 2008 sample in Essay?
On Sat, Mar 18, 2023 at 6:17 AM Jean Abou Samra wrote: > Le mercredi 15 mars 2023 à 00:57 +0100, Jean Abou Samra a écrit : > > Le mardi 14 mars 2023 à 17:44 -0600, Abraham Lee a écrit : > > At the time I started with LP, the Finale 2008 example wasn't that old > in the Essay section. Is it really fair for us to continue showing this > since quite a few versions have been released since then? I mean, Finale 27 > has been out since June 2021 and is now at 27.3. I'm not saying the > example is bad, nor doesn't it illustrate a historical > piece of evidence clarifying why LP was needed. I guess I'm wondering if > it's worth creating a new set of examples to show why it's *still* needed, > even after all these years? Thoughts? > > I suggest you open a tracker issue. > > I have done it now, https://gitlab.com/lilypond/lilypond/-/issues/6547 > Thanks, Jean, for doing that. I was hoping for a more public discussion to see if creating an issue is even warranted. The essay is a historical document, to be sure, so updating the comparison files might not be needed at all. It just feels a bit odd to read "we have chosen Finale 2008, which is one of the most popular commercial score writers". This was absolutely true... once upon a time. Reading it now makes it sound like we had to dig way back in order to pretend to make it seem like Finale isn't good enough and that LilyPond does it right. How does Finale/Sibelius/Dorico/etc. do nowadays? Do they get it right now? I'm certain folks have asked this question. For comparison, I just entered the two systems in the essay into MuseScore 4 and got a practically perfect output. Entering one voice at a time (voice 1, then voice 2), all existing pitches were maintained in voice 1 despite making alterations in voice 2 (like that omitted flat that Finale 2008 leaves out). I didn't have to correct or add anything that was missing. Maybe I just got lucky because of how I entered the passage. Arguments can be made about other layout decisions, but I think it's hard to argue against what MS4 has done compared to the hand-engraved examples: [image: image.png] So, maybe all that's needed is a different wording in this section to reflect why *at the time* this comparison made sense (like what is described at the beginning of the essay)? That would certainly be simpler than recreating the comparison (which might not come to the original conclusion like it used to). Best, Abraham
Re: Anybody else with an interest in parser wrangling?
John Wheeler writes: > On 3/19/23 11:51, David Kastrup wrote: > > When I was becoming familiar with the LilyPond manuals, it seemed to > me one manual that was missing was a concise specification of the > LilyPond language, something paralleling the R5RS for the Scheme > language. I spend a lot of time studying the lexer.ll/parser.yy code > trying to put together something along those lines for own use. Right > now that work is stuck at a *rough* draft. The notation reference at one point of time included the bison grammar. Because so much functionality has been migrated to music functions, and because the music function parsing works so much with synthetic tokens not corresponding to user input, this has been discontinued: it is not really helpful. For the purpose of parsing LilyPond files, it would be an idea to convert music functions based on their signatures into ad-hoc syntax rules and generate a syntax based on that that can be used for parsing/coloring/editing/whatever LilyPond input files. > Given that effort, I think I have a fair idea of how the lexer/parser > functions. I would be happy to contribute in this area. > > How can I be of service? I think the document to spell this out in would be the contributor's guide. It is a document of rather mixed quality and mixed coverage. But it's probably the best place to point people to anyway. -- David Kastrup
Re: Anybody else with an interest in parser wrangling?
Jean Abou Samra writes: > Le lundi 20 mars 2023 à 00:15 +0100, David Kastrup a écrit : > >> The MYBACKUP and MYPARSE stuff messes with the input in order to trigger >> syntactic decisions based on expression values. That's a bit more than >> usually expected from a Bison-generated parser. > > > Yes, I understand the basic way Bison parsers work. What I don't > understand is what other “effects” the lookahead can have, and why > having caused the reduction of the current rule is never a > problem. AFAIU, the parser works as a loop > > - Get next token from lexer. > > - Decide whether to shift or to reduce some rule. Use a lookahead > token if necessary. > > - Do the shift or the reduction and execute the semantic action. > > The lookahead token gets switched during the semantic action. Isn't it > a problem if the previous lookahead token says the current rule should > be reduced, but the new one would have required shifting? Or is that > just not a useful use of MYBACKUP/MYREPARSE? Well, if you feel you should not touch that code until you confidently know the answer to that question, let me assure you that the principal justification for that code is that I needed it to work. The equivalent in football is called a "Hail Mary pass" I believe. -- David Kastrup
Differences in `web.html`
Comparing http://lilypond.org/web.html with a self-compiled .../out-www/offline-root/Documentation/web/web.html I see the attached difference. Is it an oversight that `scripts/build/fix-docsize.sh` is not executed while building the website on 'lilypond.org'? Werner PS: That we now see 'web.pdf' instead of 'Web.pdf' seems to be a consequence of commit d638af5410936d255f.
Re: Anybody else with an interest in parser wrangling?
Le lundi 20 mars 2023 à 00:15 +0100, David Kastrup a écrit : > Jean Abou Samra <[j...@abou-samra.fr](mailto:j...@abou-samra.fr)> writes: > > > > Le dimanche 19 mars 2023 à 17:51 +0100, David Kastrup a écrit : > > > > > > > > So how to better involve others? The parser may be one of those > > > areas with an awful amount of shoestring and glue, namely fiddling > > > around until things happen to work. All that fiddling happens in > > > private before commits end up in master, meaning that it has no > > > opportunity to end up contagious the way it happens now. > > > > > > That's not really fabulous regarding the "bus factor" in that area. > > > > > > I would feel a lot more comfortable with modifying the parser if there > > was an explanation, in code comments or in the CG, of how the > > parser/lexer interplay works, when lookahead is OK or bad, and how to > > avoid it when necessary. Things like the comment above MYBACKUP > > > > ``` > > // The following are somewhat precarious constructs as they may change > > // the value of the lookahead token. That implies that the lookahead > > // token must not yet have made an impact on the state stack other > > // than causing the reduction of the current rule, or switching the > > // lookahead token while Bison is mulling it over will cause trouble. > > ``` > > > > are obscure to me. > > > Well, Bison creates LALR(1) parsers. That means that the parser always > is in a certain state. It looks at the next token, the "lookahead" > token (only one, that's what the 1 in LALR(1) is about) and then > transitions into another state while either shifting the current state > onto some stack, or by using a rule for reducing the current stack into > a production. > > The above comment is fearsome about the possibility that the > statemachine processes the current lookahead token without eating it, > but then getting the lookahead token switched out under its radar and > ending in a state that is not able to process the switched-out token. > > So far, the fears expressed in that comment have not materialized. > > The parser is only able to process a certain subset of languages. Since > the parser makes deterministic progress by either consuming a lookahead > token while growing the stack by 1 or by consuming stack material, it > ends up O(1), namely efficient with regard to the size of its input. > > When the parser applies a rule, you can specify code that will be > executed in the reduction. > > The MYBACKUP and MYPARSE stuff messes with the input in order to trigger > syntactic decisions based on expression values. That's a bit more than > usually expected from a Bison-generated parser. Yes, I understand the basic way Bison parsers work. What I don't understand is what other “effects” the lookahead can have, and why having caused the reduction of the current rule is never a problem. AFAIU, the parser works as a loop - Get next token from lexer. - Decide whether to shift or to reduce some rule. Use a lookahead token if necessary. - Do the shift or the reduction and execute the semantic action. The lookahead token gets switched during the semantic action. Isn't it a problem if the previous lookahead token says the current rule should be reduced, but the new one would have required shifting? Or is that just not a useful use of MYBACKUP/MYREPARSE? signature.asc Description: This is a digitally signed message part
Re: Anybody else with an interest in parser wrangling?
On 3/19/23 11:51, David Kastrup wrote: I've not been particularly active in the last years, and there has not really been a significant pickup in activity concerning syntax/parser. Now for better or worse, before I picked up tenure there was GLISS, the "Grand Lilypond Input Syntax Something" that sort of tried a top-down prescription of what syntax would be desirable. This suffered from a lack of feedback and input, suggestions and inspiration from the technical/implementation side and led, among other things, to a lot of churn between myself and the maintainer at that time, Graham, that ultimately contributed to him releasing the reins of the project because he figured he wasn't helping. His organisational talents that fleshed out roles and procedures working reasonably well with our scattered resources and interests of developers and documenters and other contributors can still be witnessed in the LilyPond project's somewhat unique organisational approach for getting contributions integrated and, to the degree possible, vetted. But while my desire for work on user-pointing and internal design and architecture at that time sort of unfolded mostly in a vacuum, the years since then have not seen a lot of uptake. For example, git shortlog -n -s --since '10 years ago' lily/parser.yy is sort of one-sided (admittedly, with '1 year ago' this looks different, though removing the -s argument in order to see the details also makes clear that a lot of work was not syntax related, with not much more than one minor and one more elaborate addition). As an example, I am just wrangling with extending user-definable functions in the parser, and end up with things like reduce/reduce conflicts that need to get ironed out. Bison options like -Wcounterexamples by now help with visualising what goes wrong (in my time, I used an Emacs Lisp contraption to come up with conflict examples), but one still needs to come up with plans how to circumnavigate such obstacles and it usually ends up being an exercise of trial and error a lot. There also is a lot of potential for making ping-pong progress. I realize that I am not exactly the most fun person to be working with, but also I tend to get stuck on boring or repetitive tasks to an inordinate degree. Of course it is not an inviting process to tackle those, but someone being slowed down to 20% of their potential will still easily beat someone being slowed down to 0.2% of their potential regardless of how impressive or not the respective potentials may look on paper, or more exactly before doing the gritty work of actually getting down on paper. So how to better involve others? The parser may be one of those areas with an awful amount of shoestring and glue, namely fiddling around until things happen to work. All that fiddling happens in private before commits end up in master, meaning that it has no opportunity to end up contagious the way it happens now. That's not really fabulous regarding the "bus factor" in that area. When I was becoming familiar with the LilyPond manuals, it seemed to me one manual that was missing was a concise specification of the LilyPond language, something paralleling the R5RS for the Scheme language. I spend a lot of time studying the lexer.ll/parser.yy code trying to put together something along those lines for own use. Right now that work is stuck at a *rough* draft. Given that effort, I think I have a fair idea of how the lexer/parser functions. I would be happy to contribute in this area. How can I be of service? John Wheeler
