Re: Anybody else with an interest in parser wrangling?
Jean Abou Samra writes: > Le dimanche 19 mars 2023 à 17:51 +0100, David Kastrup a écrit : >> >> So how to better involve others? The parser may be one of those >> areas with an awful amount of shoestring and glue, namely fiddling >> around until things happen to work. All that fiddling happens in >> private before commits end up in master, meaning that it has no >> opportunity to end up contagious the way it happens now. >> >> That's not really fabulous regarding the "bus factor" in that area. > > I would feel a lot more comfortable with modifying the parser if there > was an explanation, in code comments or in the CG, of how the > parser/lexer interplay works, when lookahead is OK or bad, and how to > avoid it when necessary. Things like the comment above MYBACKUP > > ``` > // The following are somewhat precarious constructs as they may change > // the value of the lookahead token. That implies that the lookahead > // token must not yet have made an impact on the state stack other > // than causing the reduction of the current rule, or switching the > // lookahead token while Bison is mulling it over will cause trouble. > ``` > > are obscure to me. Well, Bison creates LALR(1) parsers. That means that the parser always is in a certain state. It looks at the next token, the "lookahead" token (only one, that's what the 1 in LALR(1) is about) and then transitions into another state while either shifting the current state onto some stack, or by using a rule for reducing the current stack into a production. The above comment is fearsome about the possibility that the statemachine processes the current lookahead token without eating it, but then getting the lookahead token switched out under its radar and ending in a state that is not able to process the switched-out token. So far, the fears expressed in that comment have not materialized. The parser is only able to process a certain subset of languages. Since the parser makes deterministic progress by either consuming a lookahead token while growing the stack by 1 or by consuming stack material, it ends up O(1), namely efficient with regard to the size of its input. When the parser applies a rule, you can specify code that will be executed in the reduction. The MYBACKUP and MYPARSE stuff messes with the input in order to trigger syntactic decisions based on expression values. That's a bit more than usually expected from a Bison-generated parser. -- David Kastrup
Re: Musings on our translated documentation build
Il giorno lun 13 mar 2023 alle 22:20:12 +0100, Jonas Hahnfeld
ha scritto:
> I would like to see a test for a language in order to better
understand.
Here's a proof-of-concept for the French Usage Manual (sorry, if I had
correctly remembered that it was you and not Jean-Charles, I would
have
picked an Italian manual):
https://gitlab.com/hahnjo/lilypond/-/commits/poc-no-xref-maps
It works for all links inside the Usage Manual and cross-references to
other manuals (e.g. the NR), except for those to unnumbered sections
(see above). Note that external links into the French Usage Manual are
broken, the other manuals would need similar treatment.
> Would you take care of all the changes required in translated
files?
> ("swapping" @translationof and @section, changing the @ref
occurrences,
> ...)
Yes, this would be automated, I hope.
(the above proof-of-concept was mostly by hand, but I'm certainly
going
to write a script for the actual conversion...)
Hi Jonas
Actually I was curious to check the html output rather than the changes
in the source.
If it doesn't break working links, it's fine for me.
If you eventually proceed, write the scripts, etc. I'd be happy to
check if the Italian manuals look good and links work correctly. CI
builds the doc also, right? Maybe English manuals only?
Re: Anybody else with an interest in parser wrangling?
Le dimanche 19 mars 2023 à 17:51 +0100, David Kastrup a écrit : > > I've not been particularly active in the last years, and there has not > really been a significant pickup in activity concerning syntax/parser. > Now for better or worse, before I picked up tenure there was GLISS, the > "Grand Lilypond Input Syntax Something" that sort of tried a top-down > prescription of what syntax would be desirable. > > This suffered from a lack of feedback and input, suggestions and > inspiration from the technical/implementation side and led, among other > things, to a lot of churn between myself and the maintainer at that > time, Graham, that ultimately contributed to him releasing the reins of > the project because he figured he wasn't helping. > > His organisational talents that fleshed out roles and procedures working > reasonably well with our scattered resources and interests of developers > and documenters and other contributors can still be witnessed in the > LilyPond project's somewhat unique organisational approach for getting > contributions integrated and, to the degree possible, vetted. > > But while my desire for work on user-pointing and internal design and > architecture at that time sort of unfolded mostly in a vacuum, the years > since then have not seen a lot of uptake. > > For example, > > git shortlog -n -s --since '10 years ago' lily/parser.yy > > is sort of one-sided (admittedly, with '1 year ago' this looks > different, though removing the -s argument in order to see the details > also makes clear that a lot of work was not syntax related, with not > much more than one minor and one more elaborate addition). > > As an example, I am just wrangling with extending user-definable > functions in the parser, and end up with things like reduce/reduce > conflicts that need to get ironed out. Bison options like > -Wcounterexamples by now help with visualising what goes wrong (in my > time, I used an Emacs Lisp contraption to come up with conflict > examples), but one still needs to come up with plans how to > circumnavigate such obstacles and it usually ends up being an exercise > of trial and error a lot. > > There also is a lot of potential for making ping-pong progress. I > realize that I am not exactly the most fun person to be working with, > but also I tend to get stuck on boring or repetitive tasks to an > inordinate degree. Of course it is not an inviting process to tackle > those, but someone being slowed down to 20% of their potential will > still easily beat someone being slowed down to 0.2% of their potential > regardless of how impressive or not the respective potentials may look > on paper, or more exactly before doing the gritty work of actually > getting down on paper. > > So how to better involve others? The parser may be one of those areas > with an awful amount of shoestring and glue, namely fiddling around > until things happen to work. All that fiddling happens in private > before commits end up in master, meaning that it has no opportunity to > end up contagious the way it happens now. > > That's not really fabulous regarding the "bus factor" in that area. I would feel a lot more comfortable with modifying the parser if there was an explanation, in code comments or in the CG, of how the parser/lexer interplay works, when lookahead is OK or bad, and how to avoid it when necessary. Things like the comment above MYBACKUP ``` // The following are somewhat precarious constructs as they may change // the value of the lookahead token. That implies that the lookahead // token must not yet have made an impact on the state stack other // than causing the reduction of the current rule, or switching the // lookahead token while Bison is mulling it over will cause trouble. ``` are obscure to me. signature.asc Description: This is a digitally signed message part
Re: Anybody else with an interest in parser wrangling?
David Zelinsky writes:
> David Kastrup writes:
>
>> But while my desire for work on user-pointing and internal design and
>> architecture at that time sort of unfolded mostly in a vacuum, the years
>> since then have not seen a lot of uptake.
> [...]
>> There also is a lot of potential for making ping-pong progress. I
>> realize that I am not exactly the most fun person to be working with,
>> but also I tend to get stuck on boring or repetitive tasks to an
>> inordinate degree.
> [...]
>> So how to better involve others? The parser may be one of those areas
>> with an awful amount of shoestring and glue, namely fiddling around
>> until things happen to work. All that fiddling happens in private
>> before commits end up in master, meaning that it has no opportunity to
>> end up contagious the way it happens now.
>
> I've been interested in finding ways I can contribute to Lilypond,
> beyond the couple of minor merge requests I've had accepted. I don't
> know much about parsers. I read the dragon book years ago, and have
> written some seat-of-the-pants code that might qualify as parsing, for
> various things related to my math research, but nothing at all
> complicated or formal. And my experience with, say, bison, is that I've
> heard of it. :) But I'm comfortable reading documentation and happy to
> learn new things. And I have plenty of experience getting down into the
> weeds fiddling with things to get them to work just right.
>
> So if you're willing to have a bit of patience and point me in the right
> direction to learn things I need to know, I'm happy to help.
Well, I guess the first litmus test is taking a walk crossreading
lily/parser.yy and getting a feeling for what makes sense to you and
what makes your eyes glaze over.
A recent comparatively straightforward change had been
commit 276d688a358ff49e04b8b18e91ac15d56276cb62
Author: Jean Abou Samra
Date: Sun Dec 18 18:27:24 2022 +0100
Accept bare pitches/durations as music arguments to markup commands
This adds a tiny bit of the smart argument handling music functions have to
markup commands. Namely, if a command expects a Scheme argument (i.e., the
predicate is not markup? or markup-list?), and { } or {
} is
passed, the music interpretation as note event of those constructs is tried
as
well against the predicate. The most obvious use case is
\markup \rhythm { 8. }
Checking it out may be a hint about what a "comfort level" change not
involving an inordinate amount of shoestring and glue would like.
Probably the worst shoestring and glue department is how
function_arglist is destructured.
I may add some work-in-progress stuff to a branch soonish. I currently
have implemented the equivalent of define-music-function for pitches and
am mostly through with duration function (I am stuck on 4 reduce/reduce
conflicts at the moment that are essentially all the same problem).
This would warrant extending to book functions, score functions and a
few others. Most of those are likely trivial to do and don't offer a
lot (other than consistency and syntactic predictability) over
define-scheme-function.
So there are some candidates which are easy to implement in that
context, and some that are really tricky.
--
David Kastrup
Re: Anybody else with an interest in parser wrangling?
David Kastrup writes: > But while my desire for work on user-pointing and internal design and > architecture at that time sort of unfolded mostly in a vacuum, the years > since then have not seen a lot of uptake. [...] > There also is a lot of potential for making ping-pong progress. I > realize that I am not exactly the most fun person to be working with, > but also I tend to get stuck on boring or repetitive tasks to an > inordinate degree. [...] > So how to better involve others? The parser may be one of those areas > with an awful amount of shoestring and glue, namely fiddling around > until things happen to work. All that fiddling happens in private > before commits end up in master, meaning that it has no opportunity to > end up contagious the way it happens now. I've been interested in finding ways I can contribute to Lilypond, beyond the couple of minor merge requests I've had accepted. I don't know much about parsers. I read the dragon book years ago, and have written some seat-of-the-pants code that might qualify as parsing, for various things related to my math research, but nothing at all complicated or formal. And my experience with, say, bison, is that I've heard of it. :) But I'm comfortable reading documentation and happy to learn new things. And I have plenty of experience getting down into the weeds fiddling with things to get them to work just right. So if you're willing to have a bit of patience and point me in the right direction to learn things I need to know, I'm happy to help. -David
Anybody else with an interest in parser wrangling?
I've not been particularly active in the last years, and there has not really been a significant pickup in activity concerning syntax/parser. Now for better or worse, before I picked up tenure there was GLISS, the "Grand Lilypond Input Syntax Something" that sort of tried a top-down prescription of what syntax would be desirable. This suffered from a lack of feedback and input, suggestions and inspiration from the technical/implementation side and led, among other things, to a lot of churn between myself and the maintainer at that time, Graham, that ultimately contributed to him releasing the reins of the project because he figured he wasn't helping. His organisational talents that fleshed out roles and procedures working reasonably well with our scattered resources and interests of developers and documenters and other contributors can still be witnessed in the LilyPond project's somewhat unique organisational approach for getting contributions integrated and, to the degree possible, vetted. But while my desire for work on user-pointing and internal design and architecture at that time sort of unfolded mostly in a vacuum, the years since then have not seen a lot of uptake. For example, git shortlog -n -s --since '10 years ago' lily/parser.yy is sort of one-sided (admittedly, with '1 year ago' this looks different, though removing the -s argument in order to see the details also makes clear that a lot of work was not syntax related, with not much more than one minor and one more elaborate addition). As an example, I am just wrangling with extending user-definable functions in the parser, and end up with things like reduce/reduce conflicts that need to get ironed out. Bison options like -Wcounterexamples by now help with visualising what goes wrong (in my time, I used an Emacs Lisp contraption to come up with conflict examples), but one still needs to come up with plans how to circumnavigate such obstacles and it usually ends up being an exercise of trial and error a lot. There also is a lot of potential for making ping-pong progress. I realize that I am not exactly the most fun person to be working with, but also I tend to get stuck on boring or repetitive tasks to an inordinate degree. Of course it is not an inviting process to tackle those, but someone being slowed down to 20% of their potential will still easily beat someone being slowed down to 0.2% of their potential regardless of how impressive or not the respective potentials may look on paper, or more exactly before doing the gritty work of actually getting down on paper. So how to better involve others? The parser may be one of those areas with an awful amount of shoestring and glue, namely fiddling around until things happen to work. All that fiddling happens in private before commits end up in master, meaning that it has no opportunity to end up contagious the way it happens now. That's not really fabulous regarding the "bus factor" in that area. -- David Kastrup
Re: Make gitlab automatically add MR ID to commit message(s)?
Le dimanche 19 mars 2023 à 12:07 +, Werner LEMBERG a écrit : > ``` > > I think it would be beneficial if the original MR ID gets added to the > commit message(s). Is this possible? In many cases there is some > useful discussion that might help better understand code issues. > > Ideally, the 'merge request' button would do that. See [https://docs.gitlab.com/ee/user/project/merge_requests/commit_templates.html](https://docs.gitlab.com/ee/user/project/merge_requests/commit_templates.html). AFAICT, it's only possible when using merging or squash-merging, but not fast-forward merging aka rebasing. The method I use is to go to [https://gitlab.com/lilypond/lilypond/-/commits/master](https://gitlab.com/lilypond/lilypond/-/commits/master) and search for the commit's title line using the search box in the top right. For example, [https://gitlab.com/lilypond/lilypond/-/commits/master?search=convertrules.py%3A+Whitespace%2C+formatting%2C+minor+typos](https://gitlab.com/lilypond/lilypond/-/commits/master?search=convertrules.py%3A+Whitespace%2C+formatting%2C+minor+typos) If you click on the commit, you get a link to the MR where it was merged. signature.asc Description: This is a digitally signed message part
Make gitlab automatically add MR ID to commit message(s)?
I think it would be beneficial if the original MR ID gets added to the commit message(s). Is this possible? In many cases there is some useful discussion that might help better understand code issues. Ideally, the 'merge request' button would do that. Werner
Re: Benefits of Cairo backend over Ghostscript for PDF
On Sat, Mar 18, 2023 at 1:15 PM Jean Abou Samra wrote: > > It is possible to avoid this waste by using a > > graphics package implemented in pure TEX (such > > as TikZ) or using METAPOST (for which there is > > special support in dvips, dvipdfm(x) and pdfTEX to > > avoid font and glyph duplication). The package ps- > > frag doesn’t suffer from this problem either if the > > EPS files don’t contain any fonts embedded. > > *There is no need to pay attention to this tweak, > > because pdfsizeopt.py unifies font subsets.* > > I haven't tried it and don't want to (this topic is too emotional for me > right now), but if anyone is interested in reducing the size of doc PDFs when > generated with the Cairo backend, this tool could be worth exploring. See https://lists.gnu.org/archive/html/lilypond-devel/2023-01/msg00094.html You could only use this tool if you did some serious refactoring of the Cairo PDF generation. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
