Re: [RFC] Transition to Guile 3.0
On Sun, Nov 5, 2023 at 10:36 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > Hi all, > > I hear LilyPond hasn't changed its Guile version since some time (more > than 18 months). So before we get too comfortable with the current > situation, let me propose to move to Guile 3.0. Below is a plan for > that switch, with a transition period to test the official binaries. > Last time, when going from Guile 1.8 to version 2.2, the switch had to > coincide with moving away from GUB. Between Guile 2.2 and 3.0, we could > in principle support both versions for a longer period. However, I > personally think that a full transition and dropping support for Guile > 2.2 is the more reasonable approach: It will reduce testing > configurations (both for development and user reports) and hopefully > enable some future cleanups in the code. > Could you say a bit more about the benefits/disadvantages for the user? I had the impression that Guile 3 had different (worse?) performance characteristics relative to 2.2, but it's been a while. IIRC, one of the arguments to drop 1.8 is that Guile pre-2.x did not support installing multiple versions alongside each other, which forced distributions to choose whether to ship LilyPond or a recent version of Guile. With 2.2 and later, that dilemma disappeared. I quickly grepped over the source (grepping for SCM_MAJOR_VERSION), but the bifurcations look very modest. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Use of dev/ branches in Gitlab and GSOC
On Wed, May 31, 2023 at 3:56 PM David Kastrup wrote: > Han-Wen Nienhuys writes: > > > On Wed, May 31, 2023 at 5:12 AM Carl Sorensen > > > wrote: > > > >> After reading all of this, I believe I should recommend to Jason that he > >> not have his gsoc repository be on the main GitLab repository for two > >> reasons: 1) We really want the dev/ branches on GitLab to be used only > for > >> merge requests; and 2) We want the dev/ branches on GitLab to be > temporary, > >> but GSOC wants a permanent repository of the student's work. > >> > >> Am I making a mistake in giving Jason this advice? I'd welcome any > >> comments. > >> > > > > I think you are right. Creating a fork is slightly more cognitive > > overhead, but it's not prohibitive, and if GSOC wants a permanent home > > for work that is not merged, then the fork is the right place. > > I think I disagree in this particular context because the commitment > from GSOC is a temporary one, and a fork is not a "permanent home for > work that is not merged" in the GSOC context because it can just > disappear along with the original account. > > That does not mean that I am against the use of forks in general. But > for "unfinished work passing into general project reponsibility", > maintaining it under accounts with a possibly diverging interest (where > deletion is an extreme form of a diverging interest) does not appear > like the best policy to me. > To me, code passes into "general project responsibility" by being merged into the project. The requirement to keep code alive, is that something that the student agrees to with GSOC or do we agree with GSOC on this? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Use of dev/ branches in Gitlab and GSOC
On Wed, May 31, 2023 at 5:12 AM Carl Sorensen wrote: > After reading all of this, I believe I should recommend to Jason that he > not have his gsoc repository be on the main GitLab repository for two > reasons: 1) We really want the dev/ branches on GitLab to be used only for > merge requests; and 2) We want the dev/ branches on GitLab to be temporary, > but GSOC wants a permanent repository of the student's work. > > Am I making a mistake in giving Jason this advice? I'd welcome any > comments. > I think you are right. Creating a fork is slightly more cognitive overhead, but it's not prohibitive, and if GSOC wants a permanent home for work that is not merged, then the fork is the right place. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Beams in Lilypond
+list Figuring out how the number -0.63 was calculated is basically asking to read the code. Can you take a step back and explain what you need this information for? On Tue, May 30, 2023 at 6:01 AM Chad Linsley wrote: > I'm afraid my understanding of code is very limited. My hunch is that > there is an array of possible beam slopes generated which Lilypond then > calculates the least squares regression line. Is there any way to get > Lilypond to spit out this number? I can't figure out how the number -0.63 > was arrived at. > > C > > On Sat, May 27, 2023 at 1:57 AM Han-Wen Nienhuys wrote: > >> Have a look at >> https://github.com/lilypond/lilypond/blob/9af92c35746c5d6de70ae5b3af5c696741f603cd/lily/beam-quanting.cc#L539 >> >> Op za 27 mei 2023 03:22 schreef Chad Linsley : >> >>> Dear Han-Wen, >>> >>> Greetings from Montreal! I've been trying to learn about how Lilypond >>> calculates beam angles etc. and came across a great comment you made in the >>> Dorico Development diary about the subject of beams: >>> >>> "Regarding the stubby note group in your example: the scoring deems that >>> the disparity between ideal slope (-0.63 staffspace) and steinberg’s >>> example is a worse tradeoff than the shortened stems for LilyPond’s choice. >>> To be exact: the scoring for the steinberg choice is 2.57 (ideal slope) + >>> 1.57 (stem) and the LilyPond choice gets 0.17 (slope) + 2.96 (stem length)." >>> >>> https://blog.dorico.com/2015/03/development-diary-part-10/ >>> >>> With Beam.inspect-quants I was able to generate the same results of the >>> demerit points for the Steinberg and Lilypond examples as indicated. >>> However, I couldn't understand where the "ideal slope (-0.63 staffspace) >>> number comes from at the beginning of your comment. >>> >>> Anyway, thank-you for your time! >>> >>> Best wishes, >>> Chad Linsley >>> >>> >>> -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Differences in `web.html`
On Mon, Mar 20, 2023 at 3:35 PM Werner LEMBERG wrote: > > > Comparing > > http://lilypond.org/web.html > > with a self-compiled > > .../out-www/offline-root/Documentation/web/web.html > > I see the attached difference. Is it an oversight that > `scripts/build/fix-docsize.sh` is not executed while building the > website on 'lilypond.org'? note that the website isn't built on lilypond.org anymore. Rather, there is a cronjob that downloads an artifact from gitlab, see https://gitlab.com/hahnjo/lilypond-infrastructure/-/blob/master/website/main.go -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Is it time to update the Finale 2008 sample in Essay?
On Mon, Mar 20, 2023 at 5:27 PM Abraham Lee wrote: > > On Sat, Mar 18, 2023 at 6:17 AM Jean Abou Samra wrote: > > > Le mercredi 15 mars 2023 à 00:57 +0100, Jean Abou Samra a écrit : > > > > Le mardi 14 mars 2023 à 17:44 -0600, Abraham Lee a écrit : > > > > At the time I started with LP, the Finale 2008 example wasn't that old > > in the Essay section. Is it really fair for us to continue showing this > > since quite a few versions have been released since then? I mean, Finale 27 > > has been out since June 2021 and is now at 27.3. I'm not saying the > > example is bad, nor doesn't it illustrate a historical > > piece of evidence clarifying why LP was needed. I guess I'm wondering if > > it's worth creating a new set of examples to show why it's *still* needed, > > even after all these years? Thoughts? > > > > I suggest you open a tracker issue. > > > > I have done it now, https://gitlab.com/lilypond/lilypond/-/issues/6547 > > > > Thanks, Jean, for doing that. I was hoping for a more public discussion to > see if creating an issue is even warranted. The essay is a historical > document, to be sure, so updating the comparison files might not be needed > at all. I agree to this. It is better to simply put a small intro to the essay that gives the context. We could then add an updated preface/postscriptum that puts it in context for 2023. To note, a lot of modern music engravers have taken inspiration from the LilyPond attitude to engraving. The Dorico blog posts have been quite explicit about it, and maybe we could ask the MuseScore folks for comments too. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Benefits of Cairo backend over Ghostscript for PDF
On Sat, Mar 18, 2023 at 1:15 PM Jean Abou Samra wrote: > > It is possible to avoid this waste by using a > > graphics package implemented in pure TEX (such > > as TikZ) or using METAPOST (for which there is > > special support in dvips, dvipdfm(x) and pdfTEX to > > avoid font and glyph duplication). The package ps- > > frag doesn’t suffer from this problem either if the > > EPS files don’t contain any fonts embedded. > > *There is no need to pay attention to this tweak, > > because pdfsizeopt.py unifies font subsets.* > > I haven't tried it and don't want to (this topic is too emotional for me > right now), but if anyone is interested in reducing the size of doc PDFs when > generated with the Cairo backend, this tool could be worth exploring. See https://lists.gnu.org/archive/html/lilypond-devel/2023-01/msg00094.html You could only use this tool if you did some serious refactoring of the Cairo PDF generation. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: RFC: stop doing "grand replace" updates to copyright years
On Mon, Feb 13, 2023 at 11:53 AM Jean Abou Samra wrote: > > Le lundi 13 février 2023 à 11:07 +0100, Han-Wen Nienhuys a écrit : > > Hi there, > > Every year, we go over the source code to update the copyright years > that are found in the source headers. I propose to stop this. > > We started doing this because of the GNU standards which say > > https://www.gnu.org/prep/maintain/html_node/Copyright-Notices.html > > but we aren't following the instructions: the instructions are to only > update the year if there were nontrivial changes to the file. > > I don't understand this since it says > > """ When you add the new year, it is not required to keep track of which > files have seen significant changes in the new year and which have not. It is > recommended and simpler to add the new year to all files in the package, and > be done with it for the rest of the year. """ > > which sounds like exactly the opposite. I read it again, and you are right. The instructions say to update each file even if the file itself wasn't changed in that year. I guess the instructions codify what I find annoying in this practice: to touch files even if they weren't changed in any way. > If we stop doing grand-replace, does it mean we have to update the copyright > noticed manually when we change a file? No. > Git, for example, does not have an equivalent of grand-replace simply because > it does not have copyright notices in each file. > > If accepting this proposal just means no more grand-replace, I'm fine with > it, but it would seem a bit weird to keep "Copyright 1995-2023" at the top of > all files even in 2025. it is weird, but so is doing the grand update. We could decide to trim our license headers to a smaller SPDX identifier without a year, but we still have another year to go before a decision would make a difference. > To be honest, although I know there was work invested in them, I would > personally be glad to see the individual copyright notices in each file just > go away, although GNU purists will disagree. (But as a matter of fact, there > are a lot of dated recommendations in the technical side of the GNU > guidelines for maintainers.) -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
RFC: stop doing "grand replace" updates to copyright years
Hi there, Every year, we go over the source code to update the copyright years that are found in the source headers. I propose to stop this. We started doing this because of the GNU standards which say https://www.gnu.org/prep/maintain/html_node/Copyright-Notices.html but we aren't following the instructions: the instructions are to only update the year if there were nontrivial changes to the file. IIUC, this could make some difference in 95 years (?) from now when versions of the file might enter the public domain (assuming humans still exist as a species, and has a need for beautifully typeset music). I believe this advantage does not weigh up against the disadvantages, which is that it makes the output of git-log harder to read, and that the process takes up time and effort that could be more productively spent elsewhere. Also note that many other projects (eg. git) seem to survive just fine without yearly exercise like this. Also, at $dayjob, there are no instructions to do this, despite open source work being carefully overseen by lawyers. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: RFC: require librsvg to implement SVG image support
On Tue, Jan 10, 2023 at 11:32 PM Jean Abou Samra wrote: > > Le 10/01/2023 à 23:25, Han-Wen Nienhuys a écrit : > > On Mon, Jan 9, 2023 at 7:45 PM Jonas Hahnfeld wrote: > >> On Sun, 2023-01-08 at 23:18 +0100, Jean Abou Samra wrote: > >>> In order to keep support for vector graphics, even if not > >>> with EPS, we can add support for embedding SVG images. > >> Are we sure that this is actually what the users need? If everybody > >> just cares about including PDF (for logos), I'm not sure if we need to > >> implement support for SVG. > > PDF has much more features than SVG, so it's not obvious that that > > will be easier, and if we want to ingest the PDF graphic for output to > > SVG/PNG/PS, we'll have to interpret the PDF data, basically doing what > > Poppler does. > > > I don't understand how this is different from SVG. I was initially thinking there might be a way to avoid a full-blown parser/interpreter for PDF. But that would not work in all formats, so it's probably not acceptable. So you're right. For others following the conversation, the poll is at https://lists.gnu.org/archive/html/lilypond-user/2023-01/msg00189.html It's currently 3-0 skewed towards SVG. Regarding rsvg and rust: maybe we could try with an older version of librsvg first? Rust was only introduced in 2.41, so if we go with 2.40, we can postpone the worry about compiling Rust. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: RFC: require librsvg to implement SVG image support
On Mon, Jan 9, 2023 at 7:45 PM Jonas Hahnfeld wrote: > > On Sun, 2023-01-08 at 23:18 +0100, Jean Abou Samra wrote: > > In order to keep support for vector graphics, even if not > > with EPS, we can add support for embedding SVG images. > > Are we sure that this is actually what the users need? If everybody > just cares about including PDF (for logos), I'm not sure if we need to > implement support for SVG. PDF has much more features than SVG, so it's not obvious that that will be easier, and if we want to ingest the PDF graphic for output to SVG/PNG/PS, we'll have to interpret the PDF data, basically doing what Poppler does. I posted a question on stackoverflow, https://stackoverflow.com/questions/75076381/embedding-pdf-graphics-in-pdf-output-file-programmatically, maybe that will bring us some light. I looked at LuaTeX and it comes with a library for reading PDF (pplib), which looks small and self-contained. However, it may be that they can avoid looking too deeply into the PDF contents, because their output is always PDF too. However, we'd need a Cairo feature to dump PDF data directly into the output stream. > Another question would be about support in the default PS backend: Is > this feasible for SVGs? Would we again export the rendering from Cairo > and then paste into the output PS? Yes, that should not be too hard. cairo_ps_surface_create_for_stream() will write to a in-memory stream. > > This requires the ability to render an SVG to a Cairo > > surface. Cairo doesn't do this itself. It is the job of > > the librsvg library, which is part of the GNOME stack > > (like GLib and Pango which we already require). > > > > There is also PDF, which could be made to work via Poppler. > > However, it is built with CMake, for which the release > > infrastructure does not have support so far. Jonas said > > cross-compilation was difficult with CMake, and I believe > > him. > > Cross-compilation is always difficult, and what I was trying to say is > that the scripts in release/binaries/ currently don't support CMake. > Cross-compilation with CMake is sometimes a pain because IIRC it > doesn't really support the notion if different targets for host and > build, so something like a tools build for execution during the build > is not really possible. I don't know about Poppler, maybe it doesn't > have any such problems. we'd have to look. I was surprised at https://www.linuxfromscratch.org/blfs/view/svn/general/poppler.html ; the required dependencies are modest. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Missing items to make Cairo ready
On Sat, Jan 7, 2023 at 10:16 PM Jonas Hahnfeld wrote: > > > Furthermore, I'm not a fan of recommending two different ways of > > > creating PDFs to users (once directly via Cairo and once with ps2pdf), > > > unless we really, really have to. > > > > We don't really, really have to, but the advantage of dropping \epsfile > > and \postscript isn't big either, as their Cairo implementation is not > > complicated and can largely share code with the implementation of > > other image formats. It may also be useful for people who still use > > the PS/EPS output directly and not PDF output (there probably aren't > > many for PS, but EPS might be a bit more common?). > > That's not really my point; if we keep markup commands that only work > via this very specific ps2pdf conversion, we have to guarantee that > users get the same visual output as direct PDF output. Do we want to > support this? Does Cairo guarantee this for all possible cases that we > run into? I highly doubt this is a good rabbit hole to go into... > > If we don't do that the error text of \epsfile and \postscript in the > default PDF mode has to be "please use this ps2pdf conversion (which we > don't even ship with our binaries), and by the way, your PDF will look > different". That sounds horrible. The discussion was about deprecating \postscript and/or \eps-file. There are 2 different ways to understand "deprecate" 1. "We are going to remove this command shortly; use the following instructions to migrate to the supported XYZ command" 2. "We recommend against using this command in new material; use XYZ instead." I am arguing against definition 1. If we remove a command, especially an open-ended one like \postscript, we are adding a new roadblock to users that want to use the latest version of LilyPond, either because they want better typography or because they need support for newer platforms (eg. windows 64-bit, Apple silicon). The cost of adding this roadblock is high for users, while its benefit (not having an ugly error message in our code base) is small. The \postscript command is already a rabbit hole, because * it offers access to undocumented postscript commands (staff-line-thickness, draw_round_box, etc.). * it doesn't work with SVG output. * it's also a "specialist" command as writing postscript isn't for the faint of heart. However removing the rabbit-hole doesn't actually help users that have already used it in their .ly files. I am arguing that we should do what is in definition 2 (but I usually don't call this deprecate) -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Missing items to make Cairo ready
On Fri, Jan 6, 2023 at 10:19 AM Jonas Hahnfeld wrote: > > On Wed, 2023-01-04 at 12:52 +0100, Han-Wen Nienhuys wrote: > > Regarding versioning: the 1.x to 2.x transition was motivated by > > radical syntax changes that necessitated converting and 'manually' > > verifying the .ly files. Since Cairo vs. Ghostscript doesn't affect > > the semantics of .ly files, I think we can continue the 2.x version > > number. As a practical example, page layout was introduced in 2.4, and > > direct to PostScript only became default in 2.6; both changes are much > > more invasive than what we are discussing here. > > Regardless of what has been done in prior versions, it seems to me the > cleanest solution still is to remove a number of markup commands that > we cannot or do not want to support with Cairo. We know some are used > in existing libraries and scores, so this constitutes a breaking > change. What exactly is your argument for *not* going to version 3.x in > that case? I don't think there is value in removing markup commands. When we drop the PS backend, folks that use \postscript or \epsfile can do lilypond --ps foo.ly && ps2pdf foo.ps rather than lilypond foo.ly and have their scores mostly work. We don't need to remove support for this, ever. Since this doesn't break backward compatibility, I don't think we need a major version bump. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Missing items to make Cairo ready
On Sun, Jan 1, 2023 at 12:36 PM Jean Abou Samra wrote: > > Le 30/12/2022 à 13:08, Jean Abou Samra a écrit : > > which means figuring out how to do PNGs via the default PS > > backend and GS. > > > I looked a bit at this. > > It's not insurmountable, *but*, alpha transparency is not going > to work. transparency doesn't work today either in the PS backend, the stencil-expressions.ly regtest looks different in Cairo and the classic backend. See commit 103ca4c38061754f612880bcc7c29b4fd5acb8f6. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Missing items to make Cairo ready
On Thu, Jan 5, 2023 at 2:24 PM Werner LEMBERG wrote: > > > >> The only thing I would like to convince the Cairo people is to add > >> a mode to produce PDFs with font references instead of embedding – > >> and subsetting – fonts. My Cairo knowledge is zero; maybe this is > >> already possible? > > > > This makes no sense to me as a Cairo feature. Such PDF snippets > > would need custom post processing to assemble into the full > > document. > > The general problem (i.e., the inclusion of multiple PDF images into a > larger PDF document) is not unique to LilyPond. My idea is that there > exist special tools (for example, Ghostscript's 'pdfwrite' device or > `pdfsizeopt`) to produce size-optimized PDFs.[*] However, these tools > can only do their optimization job if the necessary preliminaries are > fulfilled. If the PDF images contain subsetted fonts, it doesn't work > most of the time: You either need PDFs with complete fonts (i.e., not > subsetted), or PDFs with references to fonts. I've now looked up the > Cairo manual, and it doesn't offer any control for that. > > > If we do custom post processing, we might as well postprocess the > > final PDF directly to make all snippets point to a single music font > > object? > > Font subsetting effectively prevents such post-processing since too > much information gets stripped off during this process. For the > special case of Emmentaler fonts it might work because LilyPond knows > more about these fonts than for others, but not in the general case. I played around on essay.pdf with pdfcpu. It looks like Cairo strips the encoding information when the snippet PDFs are generated, and then creates a subsetted font containing /uni0001, /uni0002 in glyph slots 1, 2, .. in the font. I was hoping we could replace the font reference within the snippet in a postprocessing step, but this is obviously impossible, as each snippet has its own encoding. It also means that the Cairo feature you propose would add a completely new way of outputting glyphs/fonts, with associated costs in coding, testing etc. >From the perspective of Cairo development, it seems like a tall order. > > IMO, working with a 35mb user manual isn't materially different from > > working with a 10mb user manual. Both take a while to download. > > Indeed, but the manuals as a whole, in all languages, get also > distributed, and there it *does* make a significant difference IMHO: > Right now, the PDFs in `lilypond-2.24.0-documentation.tar.xz` (which > has a size of 170MByte) need 144MByte in total (uncompressed). > Multiply the latter by four... Do people actually download this a lot? Unfortunately Gitlab doesn't provide this data (https://gitlab.com/gitlab-org/gitlab/-/issues/327317). I looked on lilypond.org as well, but we only have 2 weeks of data and the doc tarballs there are outdated (there were no downloads.) -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Slurs without Stems (Issue #3760)
On Wed, Jan 4, 2023 at 8:09 PM Dan Eble wrote: > > Should slurs require Stem_engraver? > > No: https://gitlab.com/lilypond/lilypond/-/issues/3760#note_339794300 > Yes: > https://gitlab.com/lilypond/lilypond/-/merge_requests/1800#note_1227831830 > > I'm looking for some input from old-timers to help decide this so I can avoid > wasting more time on broken or unacceptable solutions. I have no definite opinion either, and I created this mess ;-) In general, the formatting code tries to be lenient when Grobs are missing, and looks for fallbacks. This means that adding some support for finding the extremal head without a stem is fine, albeit that not all formatting may turn out correctly. In retrospect, I don't know if the NoteColumn grob has been a great idea: as time progressed, the original function of NoteColumn (provide the extents of a notehead + stem + flag) has been usurped by various more sophisticated mechanisms. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Missing items to make Cairo ready
On Thu, Jan 5, 2023 at 7:16 AM Werner LEMBERG wrote: > The only thing I would like to convince the Cairo people is to add a > mode to produce PDFs with font references instead of embedding – and > subsetting – fonts. My Cairo knowledge is zero; maybe this is already > possible? This makes no sense to me as a Cairo feature. Such PDF snippets would need custom post processing to assemble into the full document. If we do custom post processing, we might as well postprocess the final PDF directly to make all snippets point to a single music font object? IMO, working with a 35mb user manual isn't materially different from working with a 10mb user manual. Both take a while to download. I also wonder how people actually use the PDF manual. It's very big to print out, and compared to the HTML manual, unwieldy to use on screen. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Missing items to make Cairo ready
On Thu, Dec 29, 2022 at 1:53 AM Jean Abou Samra wrote: > > Hi, > > I have just opened issues for the missing features of > the Cairo backend that I am aware of. > > https://gitlab.com/lilypond/lilypond/-/issues/6500 > https://gitlab.com/lilypond/lilypond/-/issues/6501 > https://gitlab.com/lilypond/lilypond/-/issues/6502 > https://gitlab.com/lilypond/lilypond/-/issues/6503 > https://gitlab.com/lilypond/lilypond/-/issues/6504 > > Are there any others? I've read through the discussion you started here, but IMO we're too fixated on backward compatibility, which is unfortunate because the PS -> GS -> PDF route was never thought through carefully. We started dumping PostScript because we were going through TeX before, and creating PDF at the time involved .tex => .dvi => .ps => .pdf (with .eps included into .dvi for graphics). However, at the time, PDF rather than PS was already the common format for print documents. We were also lazy, and didn't want to bother learning how to dump PDF directly. PNG images will always be clunky for embedding line art, so it can't be the recommended solution. What makes most sense for users? Other illustration programs also can't process EPS (for the same reasons we don't), so how do those programs embed line art? What is the preferred format for logos today? SVG? AI? PDF? Given the sorry state of the SVG backend, and the niche quality of the output-attributes feature, we could be justified to drop the SVG backend rather than implementing attributes in Cairo-SVG (although it sounds like a straightforward extension to Cairo). Regarding versioning: the 1.x to 2.x transition was motivated by radical syntax changes that necessitated converting and 'manually' verifying the .ly files. Since Cairo vs. Ghostscript doesn't affect the semantics of .ly files, I think we can continue the 2.x version number. As a practical example, page layout was introduced in 2.4, and direct to PostScript only became default in 2.6; both changes are much more invasive than what we are discussing here. > I ask this because we are at an early point in the > 2.26 release cycle, which could potentially be ideal > to get testing for "Cairo by default" if it were > ready, but it isn't yet. > > Best, > Jean > -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: weird error engraving two files with 2.23.14
On Thu, Oct 13, 2022 at 1:28 PM Kevin Barry wrote: > > It works great using the cairo backend! > > I wanted to test it so I think I'll set it as default for all my scores. > > What are the current drawbacks? There's any page in the docs about cairo? > > It looks like there was some kind of mixup adding info about cairo to > the changes doc. I can see the doc code in this diff: > https://gitlab.com/lilypond/lilypond/-/merge_requests/1651/diffs?commit_id=1df07dd726ff8bf16e60501080fcbd47b82ae6d4 > but it looks like that merge request somehow didn't end up including > that. I'll see about preparing a patch for it. > > To summarise what's there: it should be an improvement over ghostscript, > but it's still experimental and doesn't implement all of the features of > the other backends (I can't remember what, if anything, is missing). Some PDF metadata features are missing, IIRC. Also, raw PostScript only works if you export to PS, but not for PDF/PNG/SVG. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Potential LSR licensing violations
On Thu, Oct 20, 2022 at 2:34 AM Jean Abou Samra wrote: > So far, so good. However, take this snippet: > > https://lsr.di.unimi.it/LSR/Item?id=102 > > It begins with 300 lines of code that used to be in the LilyPond > repository, released under the GPL, before they were considered > legacy and moved to a snippet. I am pretty sure this violates > the GPL. 300 lines looks too much for fair use law to apply, > doesn't it? Why not ask Jan who wrote the chords snippet if is he OK to relicense the snippet as public domain? (or, maybe he already did this when the snippet was submitted) -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Draft plan for next stable release
On Mon, Aug 8, 2022 at 11:06 PM Jean Abou Samra wrote: > Also, speaking to Han-Wen, if we are to have a build system freeze > around August 20, Cairo support for the official 2.24 binaries is > now or never. Unless we consider there are very good reasons for > bypassing the freeze for Cairo? I'll have a look this week. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Does scripts/auxilar/make-regtest-pngs.sh function properly?
On Sat, Jul 9, 2022 at 6:34 AM John Wheeler wrote: > I am trying to get the process documented in CG: 9.5 "Pixel-base regtest > comparison" to run. But, I am not able to get the command > > ./make-regtest-pngs.sh -j4 -o Why are you trying to run this script? David is adamant this script can't be removed (I disagree with him) because it caters for a very specific type of validation. What are you trying to validate? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Testing that no grob is created
On Fri, Jun 17, 2022 at 2:11 PM Dan Eble wrote: > > I have a regression test that should fail if a BarLine grob is created. It > is sufficient to override BarLine.glyph-name to make it obvious in the visual > diff. > > Is there a simpler, more direct, or more robust way to test this? You could add a callback in the System grob (or in a similar grouping Grob at staff level) that counts the number of bar lines included in the 'elements property. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: RFC on MR 1368
I have had many similarly exhausting discussions before, so I empathize (it is also the reason that I paused my contributions recently.) I would go with Werner's choices here; as the Freetype author, he is the expert on font features and technology. >From the MR: > I equally object to any contribution being merged "because the author knows > what he's doing". I object to reviewers blocking contributions just because they have a strong opinion on how things should be done. In this case, Jonas has made 0 contributions to the MF code, so I don't think his concerns should be overriding. If Jonas feels really strongly about how the kerning should be handled, I invite him to teach himself the joys of Metafont and try his hand at a follow-up MR. On Wed, May 25, 2022 at 12:01 AM Werner LEMBERG wrote: > > > Folks, > > > Jonas and I have an intense (and very exhausting) discussion where to > add kerning data. I want to hear more opinions whether I should go > 'route one' (which I prefer) or 'route two' (which Jonas prefers). > > Please have a look at MR 1368 > > https://gitlab.com/lilypond/lilypond/-/merge_requests/1368 > > and chime in. > > > Werner > -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Guile 3.0
On Sun, May 22, 2022 at 7:48 PM Luca Fascione wrote:
> I would like to bring up an option that I'd expect fair few of you will
> _really_ not like.
> I'm doing this not because I necessarily believe it to be a
> particularly good way forward,
> rather because I feel it is sometimes useful to articulate in words why an
> "obviously awful idea" is, in fact, awful. Or maybe it isn't
I'm missing the context for this proposal. However, here is my $0.02:
Shipping dependencies ("vendoring") can be very useful, because it
reduces the combinatorial space of version combinations that you have
to support. Ghostscript does this with a lot of its dependencies (for
example, libpng, IIRC)
Vendoring Guile seems totally impractical. The Guile compilation does
some sort of bootstrapping, which makes building it from scratch
glacially slow (like: O(1 hour)), so it would be impossible for day to
day development work.
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: C++ question on wrapper API for setting Guile fluids
On Thu, Apr 21, 2022 at 10:53 AM Jean Abou Samra wrote:
> >> On Thu, Apr 21, 2022 at 12:04 AM Jean Abou Samra
> >> wrote:
> >>> I am working on code that pervasively utilizes Guile fluids. They
> >>> should
> >>> be set in dynamic scopes.
> >> That sounds scary. Care to tell more?
> >
> > What is scary about it exactly?
I am worried about introducing problems associated with dynamic
scoping into the code base; the (un)pure stuff is already there
though, so it probably is not going to create new problems, though.
> I am trying to reimplement purity in terms of fluids, so the set of
> parameters is not hardcoded to 'start, end' and all the code doesn't
> have to be littered with functions working both as pure and impure and
> forwarding start/end to the property callbacks they
> cause. Essentially, this should look like
>
>
> {
>Dynwind_context dwc;
>dwc->make_assumption (Lily::prebreak_estimate, SCM_BOOL_T);
>... get_property (grob, "property") ...
> }
>
> or in Scheme:
>
> (under-assumptions ((*prebreak-estimate* #t))
>...)
>
> and by virtue of the dynamic context, the callback that computes the
> grob property understands that it should do pure estimates. The
> property then gets cached if it doesn't depend on assumptions, or if
> it only depends on *prebreak-estimate* (in that case with two cached
> value, prebreak and postbreak), but not if the callback uses
> *prebreak-estimate-start* or *prebreak-estimate-end*. I'll have to
> experiment with caching strategies, but this is the current idea.
OK. I'm curious to see how that turns out. Does every call to
get_property() will have to check the fluid as well to see if it
should cache the computed value?
How does the caching framework know if the computation depends on the
assumption?
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: C++ question on wrapper API for setting Guile fluids
On Thu, Apr 21, 2022 at 12:04 AM Jean Abou Samra wrote: > I am working on code that pervasively utilizes Guile fluids. They should > be set in dynamic scopes. That sounds scary. Care to tell more? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Scheme pattern for retrieving data in objects
On Fri, Apr 1, 2022 at 6:43 PM Jean Abou Samra wrote:
>
> Folks,
>
> Just a random thought in passing: I just wrote something of this kind
> once again:
>
> (let ((elements (ly:music-property m 'elements))
>(element (ly:music-property m 'element))
>(articulations (ly:music-property m 'articulations)))
>...)
>
> (let ((padding (assoc-ref details 'padding))
>(common-Y (assoc-ref details 'common-Y)))
>...)
>
> (let ((padding (ly:grob-property grob 'padding)
>(shorten-pair (ly:grob-property grob 'shorten-pair))
>(normalized-endpoints (ly:grob-property grob 'normalized-endpoints)))
>...)
>
>
> Does anyone find a value in defining a macro for this?
>
>
> \version "2.23.8"
>
> #(define-syntax-rule (fetch obj getter (sym ...) body body* ...)
> (let ((evald-obj obj)
> (evald-getter getter))
> (let ((sym (getter obj 'sym))
> ...)
> body body* ...)))
> {
>c'1
>\tweak after-line-breaking
> #(lambda (grob)
> (fetch grob ly:grob-property (left-bound-info)
> (fetch left-bound-info assoc-ref (common-Y X padding attach-dir)
> (ly:message "common-Y=~a X=~a padding=~a attach-dir=~a"
> common-Y X padding attach-dir
>\startTextSpan
>c'1\stopTextSpan
> }
It's shorter, but is it easier to understand and discover? Is the
former code (which is somewhat verbose) a real barrier to getting
coding/typesetting done?
Over the years, I've become extremely wary of syntactic sugar: it adds
an extra barrier to usage/development because everyone not only has to
learn Scheme, they also have to learn the (lilypond specific) idioms
involved.
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Future tag names for releases
On Wed, Mar 16, 2022 at 9:57 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > while trying to round up the procedure for future releases, I've come > across one point that I don't particularly like, and that are the tag > names in the git repository. For example, the latest version 2.23.6 is > tagged as release/2.23.6-1, which leads to the quite ugly and long link > https://gitlab.com/lilypond/lilypond/-/releases/release%252F2.23.6-1 > because the forward slash needs to be URL-encoded. I don't care much, but if we paint the bikeshed, can we prefix the version with a non-digit, eg. refs/tags/v2.23.6 the leading digit always has me do a double take to check it's not a number of a hex commit hash. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Sat, Feb 26, 2022 at 2:02 PM Jonas Hahnfeld wrote: > > > The same happens for C++ files, you also have to recompile. But it's > > > true that editing scm files isn't for free anymore. > > > > The Scheme compilation felt much slower, and for C++ ccache takes away > > a lot of the pain of recompiles. It also appears to be > > single-threaded? I admit not having timed it in detail. > > Yes, GUILE_AUTO_COMPILE=1 is single-threaded which is why it will never > be the default. If we precompile as part of the build, I wonder if we could use a hack like -djob-count to split lilypond into N jobs where each one does a part of the job. https://www.gnu.org/software/guile/manual/html_node/Compilation.html suggests that one can call the compiler explicitly. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Sun, Feb 27, 2022 at 10:39 AM Luca Fascione wrote: > In other words, is it approximately true that "for (almost) any real-life > score the total compilation time > is proportional to the number of NoteHeads, past a certain size"? > I'm guessing you need a few pages worth of material to kill away constant > overheads, but beyond that > is it true that if you double the source size you double the compilation time? it should be, but we have rather complicated page breaking code that is so hairy that nobody understands it fully. I'm not sure there is NP-complete snake in the proverbial grass there. >> I am not sure what you mean by effort spent >> in "preparing" callbacks, could you elaborate? > > > Imagine the grob is a C++ object that remains opaque to Scheme. So it's a > thing for which > in Scheme you move around a blind pointer, but to read property Y-offset > you'd call (grob-get 'Y-offset) > and grob-get is C++. > .. accessing data (eg. offsets): * on first access, the callback gets executed. This is just evaluating ( ). * on second access, the value is cached in an alist, and looking it up is extremely cheap. > Code following a simple pattern like this, once compiled, will largely be > dominated by the > scripting language runtime overhead >From the outside this may seem plausible, but I doubt your intuition here: * the callback tables are also alists. They are cheap (they aren't sped up if you swap them for hash tables) * Scheme has no marshaling: objects are either direct (scheme -> C++ is bit twiddling), or they are indirect (a tagged pair with a C++ pointer in the cdr) IMO The real problem is that we don't have good tooling to see what is going on in the Scheme side of things: C++ has decent perf analysis, but the Guile side of things just looks like a lot of time spent in scm_eval(). Some of it is overhead, some if it might be a poorly implemented Scheme function (which is often easier to fix than reducing overhead.) > (*) 'boxing' is a C# word to mean wrap a POD value in an object to move it > around in a way > that is uniform with the other objects in the language. C# people need to > keep boxing and > unboxing costs in their head in code with lots of PODs being moved around, > because that > cost can dominate the execution of their code. I'm not sure what word is used > in Guile for this > concept. For integers, floats and booleans, the CPU native representation involves some bit operations. Intermediate conversions could be short-cut if you have full type information, and have a sequence of those in a row. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Sun, Feb 27, 2022 at 12:31 AM Jean Abou Samra wrote: > > I never said I don't want to fix Guile 2.2 bugs, and you should know > > as I spent lots and lots of time debugging #6218. I also said I > > support moving CI to 2.2, so any MR would pass against 2.2. > > > > I am just asking to not drop 1.8 support. > > > > Most of the work we do isn't actually about Guile anyway, > > > > $ git log --since 2022/01/01 | grep ^commit|wc > > 257 514 12336 > > $ git log --grep=[Gg]uile --since 2022/01/01 | grep ^commit|wc > >7 14 336 > > I agree that the version-specific code we have (cond-expand > and GUILEV2) isn't all that much. On the other hand, I would > be glad to be able to use Guile 2 features, such as Unicode > support, when and unless, (srfi srfi-71) (let and let* > accepting multiple values), S-expression comments, > scm_c_bind_keyword_arguments, and a few others. Now > that my principal concern with the sustainability of > Guile 2 binaries (shipping bytecode with them) is cleared, > I have mixed feelings about when to leave Guile 1 behind. You say you have mixed feelings, but I think (with your updates to compilation), those feelings are ever less mixed? > In my previous post I showed that compiling all Scheme > files can be done in 20s with Guile 2 (so a few seconds > for a single file), and 4s with Guile 3 (so near instant > for one file). Would that address your concern with > compilation speed? Thanks a ton for your investigation into this. This is a game changer: MSDM-reduced 1.8: real 0m14.788s 2.2: real 0m14.648s les-nereides: 1.8: real 0m1.376s 2.2: real 0m1.224s Let's kill 1.8 support. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Sat, Feb 26, 2022 at 2:02 PM Jonas Hahnfeld wrote: > > while we make work slower for folks that work on large > > scores and can afford to side-install Guile 1.8. It also makes > > development slower for ourselves. Yes, that means some of us will > > develop on a different platform than many users. This has been the > > case since we started supporting OSX and Windows. > > I slightly disagree, running on a different platform is not the same as > running with a completely different set of dependencies. And we know > Guile 2.2 is completely different from 1.8. > > Everyone who is passionate about Guile 2.2 can develop against Guile 2.2. > > So to be clear here: You would release official binaries with Guile 2.2 > and rely on a subset of developers to fix the bugs while you make it > clear that you don't want to do that? Why would anybody accept that > job? I never said I don't want to fix Guile 2.2 bugs, and you should know as I spent lots and lots of time debugging #6218. I also said I support moving CI to 2.2, so any MR would pass against 2.2. I am just asking to not drop 1.8 support. Most of the work we do isn't actually about Guile anyway, $ git log --since 2022/01/01 | grep ^commit|wc 257 514 12336 $ git log --grep=[Gg]uile --since 2022/01/01 | grep ^commit|wc 7 14 336 -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Wed, Feb 23, 2022 at 12:36 PM Jonas Hahnfeld wrote: > > * I'm worried that introducing a new version of lilypond that is > > significantly slower than older versions creates an incentive for > > users to stay on older versions. > > > > * I grepped our source code for "guile-2" (scm) and GUILEV2, but the > > divergence of code paths seems pretty minor. Sure, it's inconvenient > > to have the odd conditional or limit ourselves to what is both in 1.8 > > and 2.2, but I'd rather see us drop 1.8 support once we see an > > obstacle that we cannot paper over. > > The problem is, in order to "see an obstacle", we always have to invest > to test every single change with Guile 1.8 and 2.2 during development, > which will be a significant time sink. > > > * The concern over CI minutes seems like it's the least important: we > > can buy more computing power (I'm happy to sponsor), and is the > > duration of CI much of a concern today? > > In the past, you've complained the loudest about slow CI builds for > merging changes... A build for 2.2 and 1.8 could be run in parallel, I think? > > I don't think we have to do the doc build across both 1.8 and 2.2, for > > example. > > I think we do because a full doc build provides coverage and stress > testing that is unachieved by the regression tests. The doc build is exactly the same as the regtest: a lot of small files that are rendered mostly in per-system mode (rather than per page). It's just more of the same, so you get better coverage for small windows of time where objects aren't GC protected. Also, you make it seem as if perfect coverage is the only alternative to scrapping the code altogether. I think that is a false dilemma. > > * We're talking about the impact of (the lack of) byte compilation on > > users, but we haven't discussed the impact on ourselves: the startup > > slowness affects our every debug/edit cycle, and byte compilation is > > impractical because we change scm files often (by virtue of checking > > out other branches). > > The same happens for C++ files, you also have to recompile. But it's > true that editing scm files isn't for free anymore. The Scheme compilation felt much slower, and for C++ ccache takes away a lot of the pain of recompiles. It also appears to be single-threaded? I admit not having timed it in detail. > > If we need to kill 1.8 support because it blocks > > something important, then so be it, but given the impact on lilypond > > development, I'd try to postpone it if possible. > > So, you propose that development continues on Guile 1.8, but we > advertise that we consider Guile 2.2 ready for production use? I don't > think that makes for a great user experience if we let them find > problems... Also "postpone" up to what point? If we drop 1.8 support now, we can drop just a tiny sliver of complexity (places marked GUILEV2 and guile-2 in C++ and Scheme respectively), while we make work slower for folks that work on large scores and can afford to side-install Guile 1.8. It also makes development slower for ourselves. Yes, that means some of us will develop on a different platform than many users. This has been the case since we started supporting OSX and Windows. Everyone who is passionate about Guile 2.2 can develop against Guile 2.2. I'm not opposed to dropping 1.8, as long as we get something substantial in return for it. For example, we might be able to drop all of the GC marking code if we defer to BDWGC for all memory management. That is an attractive proposition, but I'd like to see a prototype before we actually go there. > > * In the discussion, we've been treating "keeping GUB alive" and > > "supporting GUILE 1.8" as equivalent, but is that really the case? We > > can't have GUILE 1.8 for 64-bit windows, but for OSX/Linux, it would > > be an option with the new release scripts too? > > No, Guile 1.8 has mandatory shared libraries for some srfi modules. > There are tricks you can play to make it work with an otherwise static > build, but they are really ugly and certainly not something I think > should be done for production. ack. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Tue, Feb 22, 2022 at 12:05 PM Jean Abou Samra wrote: > > Friends, > > I don't see this thread coming to a conclusion if it stays between the same > three people, and the topic is somewhat important to LilyPond's future. More > voices would be helpful. Here are my thoughts: * GUB needs to die. Really: I don't ever want to touch that code again, and I can't fault anyone else for feeling the same. * We've got GUILE 2.2 support to a state that we can vouch for. We need to keep that state. This means we should switch CI to run on GUILE 2.2 * I'm worried that introducing a new version of lilypond that is significantly slower than older versions creates an incentive for users to stay on older versions. * I grepped our source code for "guile-2" (scm) and GUILEV2, but the divergence of code paths seems pretty minor. Sure, it's inconvenient to have the odd conditional or limit ourselves to what is both in 1.8 and 2.2, but I'd rather see us drop 1.8 support once we see an obstacle that we cannot paper over. * The concern over CI minutes seems like it's the least important: we can buy more computing power (I'm happy to sponsor), and is the duration of CI much of a concern today? I don't think we have to do the doc build across both 1.8 and 2.2, for example. * We're talking about the impact of (the lack of) byte compilation on users, but we haven't discussed the impact on ourselves: the startup slowness affects our every debug/edit cycle, and byte compilation is impractical because we change scm files often (by virtue of checking out other branches). If we need to kill 1.8 support because it blocks something important, then so be it, but given the impact on lilypond development, I'd try to postpone it if possible. * In the discussion, we've been treating "keeping GUB alive" and "supporting GUILE 1.8" as equivalent, but is that really the case? We can't have GUILE 1.8 for 64-bit windows, but for OSX/Linux, it would be an option with the new release scripts too? > No, it can't. This approach fundamentally assumes that you can run the > lilypond binary, which is not true in cross-compilation setups. If it's > about the "with-target", then you've solved a problem that doesn't > exist: Guile bytecode (at least for version 2.2) only cares about the > "cpu-endianness" and "triplet-pointer-size". As all relevant CPU > architectures of today are little-endian, and we only care about 64-bit > the bytecodes are identical (tested with a simple module, compiled for > x86_64-pc-linux-gnu, x86_64-w64-mingw32, powerpc64le-pc-linux-gnu). I'm glad to hear that, but do we know about the GUILE team's plans in this space? It would suck if they want to move to CPU dependent bytecode. > properly solve the setup for "downstreams" that apparently is now a > requirement, but at least doesn't require additional effort. I wrote I read over this thread, but I don't understand what you mean by "downstreams" here. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Pango + Cairo, text clusters & invisible glyphs
Hi there, I'm trying to fix some corner cases in LilyPond's Cairo support. For proper copy support in our PDF output (https://gitlab.com/lilypond/lilypond/-/issues/6172), I'm trying to use cairo_show_text_glyphs(). This seems to mostly work in normal cases. However, when rendering bidi text in Pango, the direction changing codes (U+202A, U+202B, U+202C) generate a cluster entry in Pango, but the glyph is invisible. Currently, LilyPond simply filters out the invisible glyph, but with cairo_show_text_glyphs that causes the UTF-8 text, glyph list and cluster mapping to go out of sync, leading to 'input clusters do not represent the accompanying text and glyph arrays' Is there a standard way to specify an invisible glyph to cairo_show_text_glyphs? I'd rather not have to edit the UTF-8 string to filter out the invisible chars. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Uploading future binaries to GitLab
On Wed, Feb 2, 2022 at 3:24 AM Jonas Hahnfeld via Discussions on LilyPond development wrote: > I hope things are coming together for a "dual" release 2.23.6 using GUB > and the infrastructure that I developed in the past months. For the > final question, I'd like to propose that we upload the binaries to > GitLab. I developed a small script which also adds direct links to the > release, see https://gitlab.com/lilypond/lilypond/-/merge_requests/1175 > > Traditionally everything has been uploaded to lilypond.org and I > believe we want to continue doing this for the source archives. For the > binaries however, that are mostly downloaded by humans and not scripts, > I think GitLab is equally good or bad (both are running on GCP), we can > still directly link to the files, and I guess Han-Wen won't be too sad > that he doesn't have to provide egress traffic for all the binaries. A big +1 to all of this. I don't care about the source archives, as downloading and building from Git directly has become much more practical over the last years, but they take up little space, so either way is fine. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Stepping down from Patch Meister role
On Sat, Jan 22, 2022 at 7:09 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > > > Finding a non-developer to fill James's shoes sounds difficult. I > > > would volunteer, but I still plan to contribute patches. > > > > git shortlog --author Lowe > > > > should be some indication that this is not a conflict of interest as > > such. It may end up a conflict of resources. > > Yes, my initial message wasn't entirely clear, it's of course possible > to contribute patches while being "Patch Meister". The "impartiality" > aspect is that you should try to apply the same standards to your own > submissions as to all other patches. > > So, there's around one week left until the end of January and this > thread has more or less died down, thus last call for volunteers. One thought: maybe we could rotate the patch meister between different volunteers? That would also help with the impartiality: if one is this week's patch meister, you could simply leave your own MRs alone. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Stepping down from Patch Meister role
On Sat, Jan 8, 2022 at 11:07 AM James wrote: > I am going to stop doing the Patch countdown at the end of this month. > > I'll continue for the next few weeks, but my last countdown will be > whatever the last date in January ends up being. > > Thanks for your understanding. Thanks for your tireless work over the many years! Your persistent shepherding certainly has helped many contributions land that would have otherwise languished. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Planning for 2.22.2
On Tue, Jan 11, 2022 at 11:35 PM Han-Wen Nienhuys wrote: > > So please let me know if you have anything that you think would be good > > to include and that I haven't backported yet. I hope wrote a comment on > > all MRs that I cherry-picked commits from, and changed the milestones > > of the corresponding issues, if there is one... > > I had a quick look at my recent changes, but you have the important > ones already. > As discussed elsewhere, it would be good to have https://gitlab.com/lilypond/lilypond/-/merge_requests/1104 in the next 2.22 release. I would do this myself, but not sure if I can just push a cherry-pick to stable/2.22 -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Planning for 2.22.2
On Mon, Jan 10, 2022 at 8:36 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > > Hi all, > > there's a number of fixes stacking up in stable/2.22 (at least after I > processed the patches that I marked for backporting in the past months) > and I'd like to tentatively plan for the next bug fix release 2.22.2 > some time in February. We could collect some bonus points by going for > the 22nd, but we'll see if and how that works out It would be awesome to release 2.22.2 on 22/2/2022 ! :) thanks for driving this! > So please let me know if you have anything that you think would be good > to include and that I haven't backported yet. I hope wrote a comment on > all MRs that I cherry-picked commits from, and changed the milestones > of the corresponding issues, if there is one... I had a quick look at my recent changes, but you have the important ones already. > I would have liked to include some more balloon fixes, but already > https://gitlab.com/lilypond/lilypond/-/merge_requests/1071 doesn't > produce the expected results (no footnote on the MultiMeasureRestNumber > and \after is not available in 2.22 at all), and the fix from > https://gitlab.com/lilypond/lilypond/-/merge_requests/1077 doesn't seem > to work either. I suspect they require the sticky grob infrastructure, > so I've given up on them unless somebody (Jean?) can tell me that which > prerequisite to pick first. I'd stick to infrastructure fixes (eg. avoid crashes), and avoid cherry-picking changes to the formatting engine -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: LilyPond | Various doc issues (!1095)
On Tue, Jan 4, 2022 at 7:55 PM Jean Abou Samra (@jeanas)
wrote:
>
> Jean Abou Samra commented on a discussion on
> Documentation/en/notation/notation-appendices.itely:
>[about InstrumentName spanners]
>
> 3289
>
> +@strong{Spanners} are a class of grobs that span up something in
>
> 3290
>
> +the horizontal direction; in most cases they contain the word
>
> On a personal note, I find the use of spanners misleading for these and would
> very much like to see them converted to breakable items like bar numbers,
> etc. That would be the main part
Converting them to breakable items adds 3 grobs per break-point, so it
has a certain cost. Since these items don't participate much in the
formatting process, there is little benefit to compensate for the
cost.
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Java finalization & smobs?
Hi Guile devs, We are debugging a GC issue in LilyPond when used with Guile 2.2, and could use some help. The issue and associated Merge Request are here: https://gitlab.com/lilypond/lilypond/-/issues/6218 https://gitlab.com/lilypond/lilypond/-/merge_requests/1035 We are using smobs with custom mark and free functions for interfacing with our C++ code. We are seeing that sometimes mark functions are called on smobs which have their dependencies already collected, leading to crashes. We can change our mark routines to avoid the crash, but it's unclear to us if this behavior is intended or not, and we worry that this will come back to bite us in the future. On the one hand, the docs for smobs state "must assume .. all SCM values that it references have already been freed and are thus invalid", which suggests that smob freeing happens in random order, which is consistent with what we see. On the other hand, Guile sets up BDWGC with GC_java_finalization=1, which should keep GC dependencies of an object alive until the object itself is finalized, and I think we have observed the mark calls that make this happen. which of the two is it? (I am seeing the problem on Fedora 35, with Guile 2.2.7 and libgc 8.0.4) -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Moving to Guile 2.2 and away from GUB
On Sat, Nov 27, 2021 at 11:27 PM Han-Wen Nienhuys wrote: > > On Fri, Nov 26, 2021 at 12:54 PM Jonas Hahnfeld wrote: > > A build without optimizations crashed more or less immediately upon > > compiling the regression tests. That said, I'm not really interested in > > debugging advice - if you have ideas, please reproduce the problem on > > your end (shouldn't be too hard) and try them. > > I tried both ideas, and it doesn't help. https://gitlab.com/lilypond/lilypond/-/merge_requests/1035 seems to help. With that change, I could run doc + doc-clean ~5 times in a row without crashes. Semi related - it looks like GUILE 2.2 is a ~30% slowdown relative to 1.8. I don't have the numbers in my head anymore; does that sound right? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Moving to Guile 2.2 and away from GUB
On Fri, Nov 26, 2021 at 12:54 PM Jonas Hahnfeld wrote: > A build without optimizations crashed more or less immediately upon > compiling the regression tests. That said, I'm not really interested in > debugging advice - if you have ideas, please reproduce the problem on > your end (shouldn't be too hard) and try them. I tried both ideas, and it doesn't help. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Moving to Guile 2.2 and away from GUB
On Thu, Nov 25, 2021 at 7:49 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > > Check out <https://gitlab.com/lilypond/lilypond/-/merge_requests/1025> > > if you can trigger this with some frequency. > > Nope, still crashes in the same place. Would be interesting to see if you can trigger it on a compile without -O2. If not, that might point to a place where we forgot to remember a value on the stack. > > Also it may be worth > > figuring out whether the crash occurs in connection with frenched > > systems (which are explicitly removed before the end of typesetting). > > How do I check this? The System that I'm marking still looks fine, as > far as I can tell, but the problem with GC issues is that it can also > be any other Grob running havoc (even though it's suspicious that I > only see it crashing in System::derived_mark...) IIRC, the whole business with grob-array skipping marking was an attempt at cutting a corner for performance and/or limiting stack depth when doing GC marking. It may very well not be relevant today, especially for BGC, which has an explicitly allocated mark stack. Does it help if you mark X/Y parents in Grob::mark_smob? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Moving to Guile 2.2 and away from GUB
On Thu, Nov 25, 2021 at 10:08 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > Fun fact: Did you try placing the compiled file (with only the > extension .go) into out/lib/lilypond/current/ccache/lily and make > lilypond load it? It results in "Unbound variable: all-grob- > descriptions", likely because the compiler doesn't understand define- > session-public which is defined in lily. Actually, this points to yet > another problem where some files (to become modules) use the module > (lily) which in turn will have to load all other modules. So we likely > have to re-think the entire layering of the scm/ directory... semi related - in order to fix the --safe mode, we'll probably have to reorganize how things depend on each other anyway (with potential incompatibilities). -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Moving to Guile 2.2 and away from GUB
On Wed, Nov 24, 2021 at 8:13 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > Assuming no major problems are found, we could then move completely to > Guile 2.2 and do releases without GUB. Both steps still require a bit > of work; I'm relatively sure there is still a GC related issue causing > (very rare) crashes with Guile 2.2 can you say more about this? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: towards a 'normal' Emmentaler font
On Mon, Nov 22, 2021 at 10:26 AM Werner LEMBERG wrote: ... > > I think (2) might be the route to go. Comments? semi-related: what about the Y dimension (depth?). For accidentals, the glyph is centered on the center of the notehead, but for integrating with text, it should typically be centered on 0.5ex ? My suggestion: make the font bboxes/advance widths work well to integrate with text, and use an auxiliary table (ie. a lisp) to specify the center of the glyph for musical purposes. I think that is your option (3). Finally, for integrating with text, should we have a different sizing scheme? (could we make it so emmentaler 20pt matches with Times New Roman 20pt?) -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: make all fails (help2man cannot find info for lilypond because lilypond cannot find libguile.so.17)
On Sun, Nov 14, 2021 at 7:03 AM Aaron Hill wrote: > NOTE: Manually specifying the library path... > > /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 \ >--library-path /opt/guile-1.8/lib/ \ >out/bin/lilypond --version > > ...works and reports "GNU LilyPond 2.23.5 (running Guile 1.8)". As > such, the build process seems to be pretty close to working. But by the > time help2man needs to run, the library path information is lost. have you tried setting LD_LIBRARY_PATH to the directory containing the .so for the locally built guile? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: strange bbox of 'flat' glyph
On Mon, Nov 8, 2021 at 7:30 PM Werner LEMBERG wrote: > >> Does anybody know why the top and the bottom of the bounding box of > >> the 'Flat' glyph, contrary to 'Sharp' and 'Natural', don't coincide > >> (more or less) with the extrema of the glyph's outline? A small > > > > I can't remember any reason. > > :-) > > The question is whether we should fix this. > > Given that such a change would only affect markups where you access > the glyph with `\flat` or something similar (that is, the markup is > handled by Pango, and we won't change the horizontal advance width), > the risk of unwanted effects like text reflow is quite low IMHO. my worry would be alignment of accidental clusters. If you tweak vertical sizes, can you still have accidentals at an octave distance without colliding? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: strange bbox of 'flat' glyph
On Sun, Nov 7, 2021 at 10:20 AM Werner LEMBERG wrote: > > > Does anybody know why the top and the bottom of the bounding box of > the 'Flat' glyph, contrary to 'Sharp' and 'Natural', don't coincide > (more or less) with the extrema of the glyph's outline? A small I can't remember any reason. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Scheme performers/translators
On Thu, Sep 30, 2021 at 12:24 AM David Kastrup wrote: > > Aaron Hill writes: > > > On 2021-09-29 12:39 pm, David Kastrup wrote: > >> The question is whether we should do something like this as default, > >> possibly conditioned on whether any acknowledgers are present? Because > >> even if we cannot react to Midi data structures (since they are not > >> Scheme-accessible for now), sometimes a translator may be enough to do > >> the trick. > > > > You say "for now" above, so is allowing Scheme to interact directly > > with the MIDI stream something that is planned? > > I have no idea what you call "planned". It is a deficiency that one > cannot use LilyPond in anything but a hardwired manner for creating > Midi. That does not mean that I know this to be in anybody's personal > work queue. At the current point of time, the Midi data structures are > not even in the Scheme memory management. I think most of them just > stick around without ever getting deleted. Audio elements get deleted in Performance::~Performance, and I'd be surprised if the MIDI stuff gets leaked. MIDI didn't show up the memory leak hunt Jonas and I did about 9 months ago. File a bug if you can repro a leak? > > Between what you have just shown and the work that already occurs in > > articulate.ly, it seems a lot of practical manipulation is possible > > without going down to the MIDI layer. So, your little patch to enable > > wider use of Scheme translators seems quite useful. At the very > > least, the more folks build on this and start playing around with the > > music, the more we would understand what a potential full Scheme > > performer support would entail. > > Well, this is not as much supporting the MIDI layer as it is employing > the translator level for messing with music during iteration. It's sort > of annoying that it doesn't work by default. It doesn't work by default because we never bothered to invest time in improving it. It's not obvious to me that a principled approach to MIDI rendering would use a broadcast/acknowledge type architecture like the typography part does. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Mon, Aug 30, 2021 at 10:49 AM Werner LEMBERG wrote: > > The current SVG backend is glacially slow, and has suffered from > > rendering discrepancies. I propose we retire it ASAP to be > > replaced by Cairo. > > I don't use SVG normally, but this sounds like a good plan, especially > because of ... > > > The Cairo SVG files are larger, but that is because they also > > embed the fonts used for texts, making the rendering exactly equal > > to the PDF/PNG. > > ... this. Jonas, aside from the discussion we had about the SVG backend's speed vs Cairo, how do you feel about replacing the SVG backend with Cairo's SVG output? That would yield more accurate SVG output at a reduced maintenance burden. The immediate motivation is that fixing cut & paste requires rearrangement of how we output text strings, and I'd rather avoid refactoring up the SVG backend if I can (https://gitlab.com/lilypond/lilypond/-/issues/6172#note_675122487 for background). This would be predicated on getting Cairo to compile on CI, in GUB and in the new compile scripts, obviously. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Reading files as byte array
On Mon, Sep 6, 2021 at 12:41 AM Lukas-Fabian Moser wrote: > > > >>> Would it be considered okay if we included a scheme function to read a > >>> binary file to a byte array? > >> GUILE already supports this out of the box. Have you seen > >> > >> https://www.gnu.org/software/guile/manual/html_node/Binary-I_002fO.html > >> > >> ? > > Yes, of course GUILE supports this. But the problem is that this does not go > > through Lilypond’s path handling. So I really do not think this would be a > > good thing to use (of course one can use that internally, but I’m talking > > about a Lilypond friendly interface). > > Just from comparing those statements: Would it be reasonable (and maybe > generally useful) to make global_path.find (used in gulp_file_to_string, > lily-guile.cc) available at scheme level? see ly:find-file. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Reading files as byte array
On Sun, Sep 5, 2021 at 3:10 PM Valentin Petzel wrote: > Would it be considered okay if we included a scheme function to read a binary > file to a byte array? GUILE already supports this out of the box. Have you seen https://www.gnu.org/software/guile/manual/html_node/Binary-I_002fO.html ? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Sat, Sep 4, 2021 at 1:14 PM Masamichi Hosoda wrote: > > > I analyzed more closely in the issue > > https://gitlab.com/lilypond/lilypond/-/issues/6172 > > > > Would you know how the mapping of character index => codepoint is > > supposed to work for this font with character 3056 (辻) ? > > Most PDF readers have a mapping > from Adobe-Japan1 CID to Unicode code points as follows. > https://github.com/adobe-type-tools/mapping-resources-pdf/blob/master/pdf2unicode/Adobe-Japan1-UCS2 Is it impossible to discover this mapping from the OTF file alone? Where does the OTF file say it is using encoding Adobe-Japan1-UCS -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Sat, Sep 4, 2021 at 9:12 AM Han-Wen Nienhuys wrote: > >find ~/.fonts -name "HaranoAjiMincho-*.otf" > > I cloned https://github.com/trueroad/HaranoAjiFontsCN > > I didn´t reallize you had many repos named HaranoAjiFontsSomething. I > installed the right ones, and can reproduce the problem. > > I'll have a look now. I analyzed more closely in the issue https://gitlab.com/lilypond/lilypond/-/issues/6172 Would you know how the mapping of character index => codepoint is supposed to work for this font with character 3056 (辻) ? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Sat, Sep 4, 2021 at 1:41 AM Masamichi Hosoda wrote: > > > I installed your Harano fonts in ~/.fonts. Fontconfig sees them, but I > > still get DroidSans when I compile the snippet. How do I reproduce the > > problem? > > Would you show me the results of the follwing commands? > > ``` > find ~/.fonts -name "HaranoAjiMincho-*.otf" > fc-list "HaranoAjiMincho" > lilypond -dshow-available-fonts 2>&1 | grep "Harano Aji Mincho" > ``` [hanwen@localhost lilypond]$ lilypond -dshow-available-fonts 2>&1 | grep "Harano Aji Mincho" family Harano Aji Mincho CN Harano Aji Mincho CN,原之味宋体 CN,Harano Aji Mincho CN ExtraLight,原之味宋体 CN ExtraLight:style=ExtraLight,Regular family Harano Aji Mincho CN Harano Aji Mincho CN,原之味宋体 CN,Harano Aji Mincho CN Heavy,原之味宋体 CN Heavy:style=Heavy,Regular family Harano Aji Mincho CN Harano Aji Mincho CN,原之味宋体 CN,Harano Aji Mincho CN SemiBold,原之味宋体 CN SemiBold:style=SemiBold,Regular family Harano Aji Mincho CN Harano Aji Mincho CN,原之味宋体 CN:style=Bold family Harano Aji Mincho CN Harano Aji Mincho CN,原之味宋体 CN:style=Regular family Harano Aji Mincho CN Harano Aji Mincho CN,原之味宋体 CN,Harano Aji Mincho CN Medium,原之味宋体 CN Medium:style=Medium,Regular family Harano Aji Mincho CN Harano Aji Mincho CN,原之味宋体 CN,Harano Aji Mincho CN Light,原之味宋体 CN Light:style=Light,Regular [hanwen@localhost lilypond]$ fc-list "HaranoAjiMincho" (empty) >find ~/.fonts -name "HaranoAjiMincho-*.otf" I cloned https://github.com/trueroad/HaranoAjiFontsCN I didn´t reallize you had many repos named HaranoAjiFontsSomething. I installed the right ones, and can reproduce the problem. I'll have a look now. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Fri, Sep 3, 2021 at 12:54 PM Masamichi Hosoda wrote:
> \version "2.22.0"
> \markup {
> \override #'(font-name . "HaranoAjiMincho")
> { 初見はハ長調で高音の白玉があった } }
> \markup {
> \override #'(font-name . "HaranoAjiMincho")
> \override #'(font-features . ("jp90"))
> { 辻井 逗子 飴玉 } }
> ```
I installed your Harano fonts in ~/.fonts. Fontconfig sees them, but I
still get DroidSans when I compile the snippet. How do I reproduce the
problem?
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Fri, Sep 3, 2021 at 11:22 AM Richard Shann wrote:
> > Thank you! Previously I just hard-coded values for A4 paper but I
> > would
> > like to do it properly this time. I notice (peeking in scm/paper.scm)
> > that there is a Scheme symbol staff-space that holds this value, but
> > I
> > don't suppose there is a documented way of retrieving this value -
> > someone writing Lilypond by hand would not need it. In practice the
> > name is good enough I guess...
>
> No sooner have I written that than I found a heap of documentation
> ("internals") in which was:
>
> staff-space (dimension, in staff space)
>
> Amount of space between staff lines, expressed in global staff-
> space.
>
> Not that this makes much sense - dimension, in staff space ... staff-
> space expressed in global staff-space?
This is normally 1.0, but if you have a book where some scores have a
differently sized staff, they will have a smaller number there.
IIRC, we scale up/down the units so they are expressed in staff space,
so if you do
(ly:output-def-lookup paper 'mm)
you'd get 1mm expressed in staff space
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Thu, Sep 2, 2021 at 8:33 PM Kevin Barry wrote: > > > This use case continues to be supported with > > Cairo. Just convert \postscript to \path, wich > > works both in the current PS backend and with Cairo. > > Is this something that can be done automatically with convert-ly? If > not then does it justify a major version bump? Postscript is a turing-complete, full fledged programming language. Specific (and common) use-cases like drawing paths can be converted (by hand!), but for a general solution, you'd have to run the command through Ghostscript, dumping to PDF and then somehow translate the PDF back in to lilypond commands. It's not completely impossible, but a lot of work. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Thu, Sep 2, 2021 at 4:32 PM Richard Shann wrote: > Thank you! I've managed to get this working, except that the border > drawn runs off the page at right and bottom. I'm now faced with the > challenge of finding out what the page size is in staff spaces - as I > presume the \path command is speaking in staff spaces while the border > box needs to be drawn within the edges of the page ... The standard staff size is 20pt, so 5pt to a staffspace, which is 1.757mm -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
On Mon, Aug 30, 2021 at 6:31 PM Jonas Hahnfeld wrote: > > Am Montag, dem 30.08.2021 um 10:16 +0200 schrieb Han-Wen Nienhuys: > > With MR 897, I have been able to run the doc build through > > Cairo. Results are very encouraging: the build is faster, while the > > resulting PDF file is smaller. > > As Werner remarked, only due to not using extractpdfmark. I'll look into PDF processing software. AFAICT, the subsetting retains the character mapping of the original font, so I think it should be possible to rewrite it to embed the Emmentaler font once, point all font references to the full font, and elide all the subsetted versions. > Giving timing for a single HTML file is a bit dubious because it > requires processing all .tely files for cross-references. For the > influence of Cairo, you really want to compare the time it takes to run > lilypond-book to get a single .texi file. However, I'd like to remark > that generating zillions of tiny snippets is not really the kind of > things users tend to do... It is the relevant metric here, because the MR enables lilypond-book processing. It is also relevant for development because our regtests/CI pipelinek are based on lilypond-book. We discussed the performance impact of Cairo earlier, but here are timings for the MSDM score, CPU pinned at 2Ghz. -dbackend=ps --ps real 0m33.043s --pdf real 0m34.241s -dbackend=cairo --ps real 0m30.772s --pdf real 0m31.245s -dbackend=svg real 2m47.836s > > Open question is how to position Cairo output and what defaults we > > should provide. > > > > * SVG. > > > > The current SVG backend is glacially slow > > IIRC the reason is the widespread use of regexes for matching glyph > nodes. I think this could be done better, and comparing speed isn't > really fair until then. I think it is a fair comparison, given that it's comparing a documented and shipping feature against an experimental feature. It would be unfair to dismiss the experimental feature due to bugs, but that's not what's happening here. The SVG backend works roughly like the PS backend (run Scheme code that translates stencils to strings, which are dumped one by one to an output file), so I am willing to bet serious money that you can't get to perform faster than the PS backend, and by extension, the Cairo backend. From a quick look, the regexp stuff could be mitigated by doing proper XML parsing, but then you'd have to add libxml bindings for GUILE to the build. However, my main point is that the SVG output is extremely slow, and in a poor state of maintenance. The missing feature (relative to Cairo) is SVG WOFF support, but: [hanwen@localhost lilypond]$ git describe HEAD release/2.20.0-1-18-g2a3a1ed05c [hanwen@localhost lilypond]$ time lilypond -dbackend=svg -dsvg-woff input/regression/les-nereides.ly GNU LilyPond 2.20.0 ... Layout output to `les-nereides.svg'.../home/hanwen/vc/lilypond/out/share/lilypond/current/scm/output-svg.scm:455:16: While evaluating arguments to ly:paper-get-font in expression (ly:paper-get-font paper (quasiquote #)): /home/hanwen/vc/lilypond/out/share/lilypond/current/scm/output-svg.scm:455:16: Unbound variable: paper After looking through our bugs, it turns out this has been broken for at least 5 years now, (see https://gitlab.com/lilypond/lilypond/-/issues/4648) I see no reason to keep the SVG backend alive given that Cairo achieves the same functionality faster and with less code. > > Here are my questions: > > > > * when could/would we drop the SVG backend? > > > > * when could/would we switch the default PS/PDF/PNG backend to cairo? > > From my current understanding of missing features, the amount of > testing the backend can have received (or rather cannot, due to > novelty), and the nature of bugs that are found (both in Cairo itself > and the integration in LilyPond), I don't think the backend is > currently in a state to be used by default. I would highly prefer to > not mix switching the default backend with switching to Guile 2.2 that > will already be disruptive enough (yes, it's going slower than I had > hoped...). I see your point, but consider the following: the PS backend represents 3500 fiddlesome Postscript and Scheme code, which is only exercised by us. With the cairo solution, the complexity of rendering and font handling is moved to Cairo, which is much more widely used and better tested. We are only left with cairo.cc which is much more straightforward. It should be expected that the cairo solution takes comparatively little effort to stabilize. > > * when could/would we drop the PS backend altogether? > > I would say that this step requires going to LilyPond 3.0, along with > removing all the features and commands that cannot be implemented in > the Cairo backend, or that we don't want to. We can discuss this in detail
Re: Cairo plans
On Tue, Aug 31, 2021 at 7:13 AM Jan Nieuwenhuizen wrote: > > Han-Wen Nienhuys writes: > > Hi, > > > Also: apologies to reviewers for the large Merge-Request. > > Unfortunately, the backend code was quite convoluted, and I didn't see > > a way except to just slash my way through it. refactoring along the > > way. > > Sounds good, can I have a look at the patch set, do you have a git > branch somewhere? I checked some projects for usage of \ps-command and > it doesn't seem to bo popular. Losing that would have been my main > concern, I think. it's here, https://gitlab.com/lilypond/lilypond/-/tree/dev/hanwen/cairo-lp-book You can download with git fetch https://gitlab.com/lilypond/lilypond.git dev/hanwen/cairo-lp-book:foo -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo plans
+jan for real now. Also: apologies to reviewers for the large Merge-Request. Unfortunately, the backend code was quite convoluted, and I didn't see a way except to just slash my way through it. refactoring along the way. On Mon, Aug 30, 2021 at 10:16 AM Han-Wen Nienhuys wrote: > > With MR 897, I have been able to run the doc build through > Cairo. Results are very encouraging: the build is faster, while the > resulting PDF file is smaller. Also, I think we could skip emitting > EPS files for Cairo altogether, which would be another small speedup. > > In timings below, I started with an empty out-www/ and lybook-db/ directory, > > PS backend (gs-api, no extractpdfmark) > > $ time make -j4 out=www out-www/en/notation.pdf > > real 1m44.958s > user 5m26.919s > sys 0m14.347s > > $ time make -j4 out=www out-www/en/notation-big-page.html > real 2m15.521s > user 7m34.088s > sys 0m19.567s > > $ ls -lh out-www-trad/en/notation.pdf > -rw-r--r--. 1 hanwen hanwen 38M Aug 30 08:48 out-www-trad/en/notation.pdf > > Cairo > > $ time make -j4 out=www out-www/en/notation.pdf > real 1m20.922s > user 4m4.571s > sys 0m9.407s > >$ time make -j4 out=www out-www/en/notation-big-page.html > real 1m39.933s > user 5m33.227s > sys 0m12.100s > > $ ls -lh out-www-cairo/en/notation.pdf > -rw-r--r--. 1 hanwen hanwen 13M Aug 30 08:45 out-www-cairo/en/notation.pdf > > If you want to reproduce this, download MR 897, and apply this to > force Cairo: > > diff --git a/scripts/lilypond-book.py b/scripts/lilypond-book.py > index 2d5b9d76b9..33c570c9c6 100644 > --- a/scripts/lilypond-book.py > +++ b/scripts/lilypond-book.py > @@ -668,6 +668,7 @@ def main(): > > global_options.process_cmd += ( > ' '.join([' -I %s' % mkarg(p) for p in global_options.include_path]) > ++ ' -dbackend=cairo ' > + ' -daux-files ') > > global_options.formatter.process_options(global_options) > > for size comparison, notation.pdf for 2.22 is 6.7M > > For the final website, we anti-alias the images by generating them at > twice the resolution and then scaling them down. This seems > unnecessary with the Cairo output: the rendering already uses > antialiasing, and IMO, the plain Cairo output looks better than > antialiased GS (images attached). > > Open question is how to position Cairo output and what defaults we > should provide. > > * SVG. > > The current SVG backend is glacially slow, and has suffered from > rendering discrepancies. I propose we retire it ASAP to be > replaced by Cairo. The only feature missing is the 'svg-woff option; > not sure how important that is? (CC'ing Jan who implemented this.) > > [hanwen@fedora lilypond]$ time lilypond --svg > input/regression/les-nereides.ly > GNU LilyPond 2.23.4 > .. > real 0m1.662s > [hanwen@fedora lilypond]$ time lilypond --svg -dbackend=cairo > input/regression/les-nereides.ly > GNU LilyPond 2.23.4 > > real 0m0.728s > > The Cairo SVG files are larger, but that is because they also embed > the fonts used for texts, making the rendering exactly equal to the > PDF/PNG. > > * PS/EPS > > The Cairo backend doesn't support \ps-command. This is unavoidable, > and probably a feature rather than a bug. It also doesn't support > \eps-file, but that can be made to work: see > https://www.cairographics.org/manual/cairo-PostScript-Surfaces.html#eps) > > However, if Cairo is going to be the default, we should rather think > about PDF first (ie. embedding PDF images into the music). That's > possible, but we'd need to link in Poppler: > https://www.cairographics.org/cookbook/renderpdf/ > > The PS backend also has support for a couple of options that Cairo > doesn't have > > - embed-source-code > - font-ps-resdir > - gs-load-fonts > - gs-load-lily-fonts > - gs-never-embed-fonts > - music-font-encoding > - outline-bookmarks > > I think we can't support the options that tweak font loading, but do > we need to, if we generate our docs directly from PDF, and the result > is reasonably small? > > Dropping Ghostscript altogether would let us delete ~5000 lines of > code, simplify runtime (no more subprocessing), our build (don't have > to build GS), and simplify our licensing story (no more potential AGPL > dependency). > > Here are my questions: > > * when could/would we drop the SVG backend? > > * when could/would we switch the default PS/PDF/PNG backend to cairo? > > * when could/would we drop the PS backend altogether? > > Thoughts? > > -- > Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Cairo plans
With MR 897, I have been able to run the doc build through Cairo. Results are very encouraging: the build is faster, while the resulting PDF file is smaller. Also, I think we could skip emitting EPS files for Cairo altogether, which would be another small speedup. In timings below, I started with an empty out-www/ and lybook-db/ directory, PS backend (gs-api, no extractpdfmark) $ time make -j4 out=www out-www/en/notation.pdf real 1m44.958s user 5m26.919s sys 0m14.347s $ time make -j4 out=www out-www/en/notation-big-page.html real 2m15.521s user 7m34.088s sys 0m19.567s $ ls -lh out-www-trad/en/notation.pdf -rw-r--r--. 1 hanwen hanwen 38M Aug 30 08:48 out-www-trad/en/notation.pdf Cairo $ time make -j4 out=www out-www/en/notation.pdf real 1m20.922s user 4m4.571s sys 0m9.407s $ time make -j4 out=www out-www/en/notation-big-page.html real 1m39.933s user 5m33.227s sys 0m12.100s $ ls -lh out-www-cairo/en/notation.pdf -rw-r--r--. 1 hanwen hanwen 13M Aug 30 08:45 out-www-cairo/en/notation.pdf If you want to reproduce this, download MR 897, and apply this to force Cairo: diff --git a/scripts/lilypond-book.py b/scripts/lilypond-book.py index 2d5b9d76b9..33c570c9c6 100644 --- a/scripts/lilypond-book.py +++ b/scripts/lilypond-book.py @@ -668,6 +668,7 @@ def main(): global_options.process_cmd += ( ' '.join([' -I %s' % mkarg(p) for p in global_options.include_path]) ++ ' -dbackend=cairo ' + ' -daux-files ') global_options.formatter.process_options(global_options) for size comparison, notation.pdf for 2.22 is 6.7M For the final website, we anti-alias the images by generating them at twice the resolution and then scaling them down. This seems unnecessary with the Cairo output: the rendering already uses antialiasing, and IMO, the plain Cairo output looks better than antialiased GS (images attached). Open question is how to position Cairo output and what defaults we should provide. * SVG. The current SVG backend is glacially slow, and has suffered from rendering discrepancies. I propose we retire it ASAP to be replaced by Cairo. The only feature missing is the 'svg-woff option; not sure how important that is? (CC'ing Jan who implemented this.) [hanwen@fedora lilypond]$ time lilypond --svg input/regression/les-nereides.ly GNU LilyPond 2.23.4 .. real 0m1.662s [hanwen@fedora lilypond]$ time lilypond --svg -dbackend=cairo input/regression/les-nereides.ly GNU LilyPond 2.23.4 real 0m0.728s The Cairo SVG files are larger, but that is because they also embed the fonts used for texts, making the rendering exactly equal to the PDF/PNG. * PS/EPS The Cairo backend doesn't support \ps-command. This is unavoidable, and probably a feature rather than a bug. It also doesn't support \eps-file, but that can be made to work: see https://www.cairographics.org/manual/cairo-PostScript-Surfaces.html#eps) However, if Cairo is going to be the default, we should rather think about PDF first (ie. embedding PDF images into the music). That's possible, but we'd need to link in Poppler: https://www.cairographics.org/cookbook/renderpdf/ The PS backend also has support for a couple of options that Cairo doesn't have - embed-source-code - font-ps-resdir - gs-load-fonts - gs-load-lily-fonts - gs-never-embed-fonts - music-font-encoding - outline-bookmarks I think we can't support the options that tweak font loading, but do we need to, if we generate our docs directly from PDF, and the result is reasonably small? Dropping Ghostscript altogether would let us delete ~5000 lines of code, simplify runtime (no more subprocessing), our build (don't have to build GS), and simplify our licensing story (no more potential AGPL dependency). Here are my questions: * when could/would we drop the SVG backend? * when could/would we switch the default PS/PDF/PNG backend to cairo? * when could/would we drop the PS backend altogether? Thoughts? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: GUB / ghostscript
On Mon, Aug 9, 2021 at 12:23 PM Knut Petersen wrote: > > Hi everybody, > > is there any reason that we still use very ancient versions of ghostscript > for our installers prepared via GUB? in gub, there is a comment. # ***FIXME*** Ghostscript 9.20 for freebsd-x86 raises seg fault. at some point, Ghostscript moved from the GPL to the AGPL license, but I can't discover the exact version. AFAICT, 9.20 is still available under GPL though. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Hosting a FUDforum on lilypond.org
On Mon, Aug 2, 2021 at 2:25 PM Jean Abou Samra wrote: > I was able to install it on a local Apache server. In my > experiments, I encountered some issues when trying to post to > the lists via it [13], but only with one SMTP server; another > was fine. The rest of the functionality worked correctly. Thus, > FUDforum appears to be a potential solution. And I do not see > other solutions currently. I therefore propose to try it out > for real. > > The big question is: where to host it? There is lilynet.net, > administrated by Valentin, but he's been busy in the past > year, so he might not want to take that blame. Before I buy > a domain, how about lilypond.org? That would make it more > of an official community space and more discoverable for > newcomers. > > Han-Wen (administrator of lilypond.org if I understand > correctly), and all, what do you think? If someone wants to take on the task of administering a forum, I could setup a DNS entry for forum.lilypond.org. However, the fact that all these forums have fallen over also holds a lesson: hosting content is work, because it needs moderation, and dealing with all the things that can go wrong when you have users (hijacked accounts, spam, GDPR/cookie wall etc.). Also, the security record of fudforum doesn't have me impressed: * https://www.exploit-db.com/exploits/40803 * https://www.cybersecurity-help.cz/vdb/SB2019111508 The facebooks and reddits of this world are experts at running messaging platforms. Should we really spend precious developer time on running a message board? I know I won't. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo: Please test mingw installer
On Thu, Jul 29, 2021 at 11:32 AM Michael Käppler wrote: > Hi Knut, > thank you very much for your work! > > Am 29.07.2021 um 01:13 schrieb Knut Petersen: > > Here are installers for lilypond with the cairo backend after a > > significant remix by Han-Wen. GUB has been changed to use a cairo > > version that fixes three bugs relevant for lilypond, a test would be > > nice. > > > Here the test results that I could acquire so far. > At first I tested the mingw installer under Windows 10. > > The first problem I noticed with a bigger score was that Lilypond failed > with > "schwerer Fehler: Cairo context status 'invalid tag name, attributes, or > nesting'" > > This happens only if point-and-click is enabled. Otherwise it compiles > fine. > Under linux-64, with the same input file, this error does not occur. > Other files also work fine under both architectures. > Unfortunately I do not have time to cook a minimal failing example. > > The following observations are tied to the mingw installer. > > A small nit: SVG files are the only ones having no additional 'cairo' > file extension. Is this intended? > "-dbackend=cairo" has to be last. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Is there any use for the 'char' stencil command?
On Sun, Jun 27, 2021 at 8:11 PM Knut Petersen wrote: > > Is the char stencil really an unused artifact that might be removed looks like it. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Cairo
On Sat, Jun 26, 2021 at 10:52 PM Jonas Hahnfeld via Discussions on
LilyPond development wrote:
>
> Am Samstag, dem 26.06.2021 um 19:43 + schrieb Werner LEMBERG:
> > > No, it's definitely too soon for a merge request. A few
> > > stencils->cairo procedures need to be added, a bit of error
> > > handling
> > > (to prevent at least possible segfaults) is necessary, a bit of
> > > logical cleanup is desirable, etc.
> >
> > Well, you can start the commit message of the merge request with
> > 'WIP:'; this prevents execution of the CI stuff, AFAIK.
>
> No, it doesn't. There are ways to not run CI jobs for a particular
> commit, but I don't really see the point just for the sake of pushing
> the current demonstrator into a merge request.
>
> May I instead propose an incremental approach? Implementing a fully
> capable Cairo backend probably won't happen in a single MR, and making
> it the default will be even further down the road. I think for a first
> step the Cairo stuff needs to be optional and in a proper backend, not
> hacked into the current Postscript code. Then others can test, missing
> features can be added over time, and it's possible to evaluate further
> steps.
I agree. Creating a new backend is actually fairly simple. Just copy
framework-ps.scm to framework-cairo.scm. That should be enough to get
you support for -dbackend=cairo. If you want, you can also copy
output-ps.scm, but that's not strictly necessary (anything that
populates the dispatch alist for the paper-outputter will work.)
I had a quick look at the C++ code. The global variables are a bit
concerning. As a start, you could put them into
class Cairo_context {
RefPtr cr;
...
}
this struct will then later be a natural candidate to create a smob
from. However, I wonder why you want to go through Scheme in the first
place, as the output mechanism (Cairo) lives in C++, while steering
the output goes through C++ as well (see paper-outputter.cc).
For the prototype, I think it would be better to insert the Cairo data
(Cairo::Context etc.) into Paper_outputter itself. Then, you can
intercept the Stencils directly in Paper_outputter::output_scheme.
Once that works, we can figure out something with inheritance to
factor out the cairo-specific code.
Hope this helps,
Han-Wen
>
> Jonas
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: State of LilyPond with Guile 2.2
On Mon, Apr 12, 2021 at 9:36 AM Jonas Hahnfeld wrote: > > Am Montag, dem 12.04.2021 um 09:28 +0200 schrieb Han-Wen Nienhuys: > > Not being able to use 64-bit addressing on Windows with GUILE 1.8 is > > an extremely serious problem. What is the reason for this? Is it > > because dynamic loading doesn't work correctly, and GUILE tries to > > load SRFI modules as .dlls ? > > No, the dynamic loading works on Windows (after all, how would 32 bit > work otherwise?) The issue is that 64 bit on Windows uses a very weird > ABI and long's are 4 bytes whereas pointers are 8 bytes. Guile isn't > prepared to handle this, and the integrated GC fails in wonderful ways > on this issue. If you want to know more, there's quite a bit of > material just one search away. I did a search, but clearly not the right one :) - do you have some pointers? > Not sure if I mentioned this last year, > but I actually tried to fix in Guile and gave up after a week or so. I > seriously don't think that fixing this ourselves when a solution via > bdwgc is readily available is a good investment of resources. I had a look at GUILE's git history, and unfortunately, the BDGCW change is one that was entangled with lots of other changes in the run-up to 2.0. Sigh. I am quite familiar with the 1.8 garbage collector, so I'm curious where the fundamental problems come from, but given the traction of the fixes I sent to TTN, investigating this for real would only be an option if we went with the 1.8 fork route. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: State of LilyPond with Guile 2.2
On Sun, Apr 11, 2021 at 7:55 PM Werner LEMBERG wrote: > > Guile 2.2 also makes binary distribution much nicer (because there > > no more shared srfi libraries, so lilypond can be linked as one > > static executable) and makes it possible to offer 64 bit executables > > for Windows. > > ... especially to support some 64bit architectures. > > > But given the reactions, I'll reduce activity on my work towards > > Guile 2.2... > > Hopefully there is not a misunderstanding: I very much appreciate your > activities (and I'm quite sure that Han-Wen thinks the same) and > invaluable tests! > > It is very unfortunate that more recent Guile versions cause such a > serious deterioration for LilyPond. Maybe it makes sense for the > foreseeable future to stay with the status quo, this is, using Guile > 1.8 as much as possible, and offering support for 2.x (or 3.x) for > platforms where version 1.8 can't be used. And if we go this route it > makes sense to do what Han-Wen suggests, namely to add Guile 1.8 as a > submodule. Jonas didn't include 3.0 in the benchmarking because it's not generally available yet, but I think it is relevant data. Part of the reason why we want to be on an up-to-date release of GUILE is the promise that future releases will be better for LilyPond. By looking at 3.0, we can predict what the user experience will be in a few years. I am mainly worried that GUILE will further evolve into an optimizing compiler system, where the compilation takes more time, the interpreted code runs more slowly, and the build process creates increasingly complicated requirements. In this sense, the move to GUILE 2+ is a watershed moment: once we drop support for 1.8 (dropping GC glue), it will be really hard to go back, and we basically sign up to follow GUILE in their roadmap. By contrast, GUILE 1.8 is mostly C, so there is a fair chance that we can fix problems we find by ourselves. It seems that most of the core work on GUILE is done by a single person (Andy Wingo). If he leaves the project for one reason or another, is there anyone who can productively get things done on the GUILE codebase? I don't think I am confident about becoming versed in Scheme enough to tinker with an optimizing compiler written in a dynamically typed language. Not being able to use 64-bit addressing on Windows with GUILE 1.8 is an extremely serious problem. What is the reason for this? Is it because dynamic loading doesn't work correctly, and GUILE tries to load SRFI modules as .dlls ?
Re: State of LilyPond with Guile 2.2
On Sun, Apr 11, 2021 at 4:24 PM Jonas Hahnfeld wrote: > > Am Sonntag, dem 11.04.2021 um 16:04 +0200 schrieb Han-Wen Nienhuys: > > On Sun, Apr 11, 2021 at 3:42 PM Jonas Hahnfeld via Discussions on > > LilyPond development wrote: > > > > > > All numbers are from my laptop running Arch Linux (with pango > > > downgraded to 1:1.48.2-1 to keep out the memory hogging in 1.48.3) > > > and > > > measured with "/usr/bin/time -v". I use commit fce156f219 from > > > https://gitlab.com/lilypond/lilypond/-/merge_requests/723 (see > > > User's > > > View) and the system-provided versions of Guile 1.8.8 and Guile > > > 2.2.6. > > > I don't test Guile 3.0 because it's not available as package, and > > > there > > > seem to be more changes needed to make it even work. > > > I configured the build with --enable-gs-api (because I'm interested > > > in > > > the time spent in LilyPond, not how fast the system can fork gs) > > > > > > > I assume this is also without extractpdfmark, which is also a CPU > > hog. > > Correct, all numbers are without extractpdfmark. > > > > The first set of experiments is about things that developers care > > > about, namely 'make test' and 'make doc'. For all runs, I used the > > > options "-j4 CPU_COUNT=4". > > > > > > 'make test' when compiled with Guile 1.8.8: > > > * User time (seconds): 520.68 > > > * Elapsed (wall clock) time (h:mm:ss or m:ss): 3:59.54 > > > > > > > IIRC, on my 4 CPU machine, the actual time running the regression .ly > > files is fairly low (about 1 minute). A lot of time is spent running > > the lilypond-book regression tests, which makes this difference all > > the more dramatic > > True, but the current way of testing lilypond-book is kind of the > worst-case scenario for LilyPond because all files are compiled > separately. So 37 times a startup time of 2 seconds is already the > majority of the slowdown. This may warrant some further investigation. Ideally, the shared snippet database makes it so we have to run lilypond only a few times. > > > * Maximum resident set size (kbytes): 372560 > > > 'make test' when compiled with Guile 2.2.6: > > > * User time (seconds): 710.35 > > > * Elapsed (wall clock) time (h:mm:ss or m:ss): 5:43.04 > > > * Maximum resident set size (kbytes): 372344 > > > > > > -> 40% slower when using Guile 2.2.6 during development > > > > > > > > > I didn't see you mention this explicitly: for apples to apples > > comparisons, you have to disable 'debug-gc-object-lifetimes feature. > > It causes a significant slowdown with Guile 1.8 for make test/make > > doc, so the comparison is probably worse. > > > > This comes at the expense of increased memory usage (which reduces GC > > overhead), so you'd have to test with my GUILE patches for a > > completely honest comparison. I think the advantage of the decreased > > GC overhead is minor, though. > > Right, I didn't think about this. However, I compiled without --enable- > checking so the actual difference is just garbage collection happening > between the files, which you think should be minor. Also "apples to > apples" comparison means without your patches, because that's what > users get. That's not what I meant. The GC between the files is the expensive part, because it is done unconditionally and has to mark all the reachable data, and sweep the entire heap. Dropping that (according to commit e36b7c7ea98) is a 20% speedup, which we already collected for GUILE 2.2, but not for 1.8. Due to the heap scaling bugs, removing the GC between files increases the heap size with 1.8, and if you don't force GC, for a heap that is 5x larger, you run GC 5x fewer times. The GC runs themselves involve a larger heap, so this evens out, but you have to mark the reachable set 5x fewer times. This is what I meant with a 'minor' advantage. > > > In my opinion, the numbers show that we *must have* compiled > > > bytecode > > > for user installations in order to reach acceptable performance > > > (memory > > > usage seems to be fine). And while compilation by invoking LilyPond > > > is > > > somewhat odd, it works and would be viable for the beginning. > > > > > > For development, I'm less convinced. Sure, 'make test' and 'make > > > doc' > > > get faster but the compilation itself takes a considerable amount > > > of > > > time. Moreover it is my understanding the Guile is notoriously bad > > >
Re: State of LilyPond with Guile 2.2
On Sun, Apr 11, 2021 at 3:42 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > > All numbers are from my laptop running Arch Linux (with pango > downgraded to 1:1.48.2-1 to keep out the memory hogging in 1.48.3) and > measured with "/usr/bin/time -v". I use commit fce156f219 from > https://gitlab.com/lilypond/lilypond/-/merge_requests/723 (see User's > View) and the system-provided versions of Guile 1.8.8 and Guile 2.2.6. > I don't test Guile 3.0 because it's not available as package, and there > seem to be more changes needed to make it even work. > I configured the build with --enable-gs-api (because I'm interested in > the time spent in LilyPond, not how fast the system can fork gs) > I assume this is also without extractpdfmark, which is also a CPU hog. > The first set of experiments is about things that developers care > about, namely 'make test' and 'make doc'. For all runs, I used the > options "-j4 CPU_COUNT=4". > > 'make test' when compiled with Guile 1.8.8: > * User time (seconds): 520.68 > * Elapsed (wall clock) time (h:mm:ss or m:ss): 3:59.54 > IIRC, on my 4 CPU machine, the actual time running the regression .ly files is fairly low (about 1 minute). A lot of time is spent running the lilypond-book regression tests, which makes this difference all the more dramatic > * Maximum resident set size (kbytes): 372560 > 'make test' when compiled with Guile 2.2.6: > * User time (seconds): 710.35 > * Elapsed (wall clock) time (h:mm:ss or m:ss): 5:43.04 > * Maximum resident set size (kbytes): 372344 > > -> 40% slower when using Guile 2.2.6 during development > I didn't see you mention this explicitly: for apples to apples comparisons, you have to disable 'debug-gc-object-lifetimes feature. It causes a significant slowdown with Guile 1.8 for make test/make doc, so the comparison is probably worse. This comes at the expense of increased memory usage (which reduces GC overhead), so you'd have to test with my GUILE patches for a completely honest comparison. I think the advantage of the decreased GC overhead is minor, though. In my opinion, the numbers show that we *must have* compiled bytecode > for user installations in order to reach acceptable performance (memory > usage seems to be fine). And while compilation by invoking LilyPond is > somewhat odd, it works and would be viable for the beginning. > > For development, I'm less convinced. Sure, 'make test' and 'make doc' > get faster but the compilation itself takes a considerable amount of > time. Moreover it is my understanding the Guile is notoriously bad at > determining which files to recompile, in particular when macros are > involved. > > Personally, I would be ok with the moderate slowdown if that was the > only thing preventing a hypothetic switch to Guile 2.2, and the arising > What does "switch to" mean precisely? Do you mean "remove support for guile 1.8", "make 2.2 the default (when given the choice) but keep supporting 1.8" ? > question is really a matter of prioritizing: There are other items that > need solutions before that switch could happen (in particular the > release process for binary builds and documentation). What do others > think? Or would you say that proper bytecode compilation is required > before moving to Guile 2.2? (with no clear estimate how feasible that > is and how long it would take) > How hard is it really to byte-compile the files during the build? Could we run lilypond during the compile and collect the .go files? Are they architecture independent (could we still cross-compile to windows?) Sorry for the long and densely packed message, and thanks for reading > to the end! > no worries, thanks for taking the effort to sort this out! -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Test reader speed of Guile 3.0.6
On Sat, Mar 13, 2021 at 11:30 PM Dr. Arne Babenhauserheide
wrote:
> Also Guix is helping to push Guile towards the requirements of Lilypond
> again — see
> https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile
>
> Can I forward your results to the Guile mailing list?
Yes.
> > The evaluation speed of GUILE 3.x is still pretty poor. Having fast,
> > JIT'ed code seems interesting in theory, but the way it's implemented
> > in Guile 3.x is a giant headache: the separate byte compilation is
> > extremely slow, and it is hard to manage (where should the .go files
> > be stored/installed, how/when are they generated etc.). It also
> > doesn't match our use case, because a lot of the code that we have
> > comes from .ly files, so it cannot be precompiled.
>
> The article linked above shows that setting -O1 as optimization of the
> code could help (if you’re not already doing that).
The article gives a pointer to the code, which is
module/language/tree-il/compile-bytecode.scm. I ran
for c in $(git log module/language/tree-il/compile-bytecode.scm|grep
commit|awk '{print $2;}'); do git show $c ; done | grep -i doc
to look for documentation, but couldn't find it. The module has one
exported symbol, which is compile-bytecode.
Could you give some practical tips on how we'd use this?
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Test reader speed of Guile 3.0.6
On Fri, Mar 12, 2021 at 11:28 PM Dr. Arne Babenhauserheide wrote: > >> there’s a Guile 3.0.6 release planned that includes a rewrite of the > >> reader in Scheme. It has speed in the same order of magnitude as the > >> previous reader but might have different performance characteristics. > >> > >> If I remember correctly, lilypond uses the reader a lot, so if you have > >> a test-system with lilypond on Guile 3, could you test how running > >> lilypond with the current Guile master from git affects lilypond? > > > > last time I looked, building GUILE 3 from source was truly glacial, > > making this kind of thing annoying to check. > > If you build from tarball it is much faster, because it then provides > pre-created bootstrapping files. What’s so slow is creating the initial > optimized reader. That wouldn't work for testing a prerelease, though. > > > You say "same order of magnitude". Do you have benchmarks so we know > > what to expect? > > The current *average* spead of the reader is roughly 80% of the reader > implemented in C, but with different performance characteristics. I’m $ cat ../bench.ly #(define (microseconds) (let* ((t (gettimeofday)) (us (/ (cdr t) 100.0))) (+ (car t) us))) #(define start (microseconds)) % \include "bench-largeexp.ly" \include "bench-manysmall.ly" #(newline) #(display (- (microseconds) start)) Parsing & evaluating '(1 2 3) 200 times. - guile 1.8: 1.25ms - guile 2.2: 3.2ms - guile 3.0.6: 2.08ms Parsing & evaluating the giant expression in define-grobs.scm once: - guile 1.8: 10.6ms - guile 2.2: 166ms - guile 3.0.6: 71ms Parsing & evaluating the giant expression in define-grobs.scm once (but quoted, ie. not real evaluation): - guile 1.8: 10.0ms - guile 2.2: 13ms - guile 3.0.6: 12.8ms In summary, the read speed itself for large expressions is on the same order as 1.8, but for many small expressions (which is a much more common use-case) there is still a 60% slowdown. > asking here because I want to avoid surprising and avoidable changes > that block Lilypond. I consider Lilypond to be the most important > flagship project of Guile, and I want to do what I can to prevent > unnecessary friction. I appreciate the heads up you gave here today, but from our side, it doesn't seem like the Guile project is much concerned with our needs. The evaluation speed of GUILE 3.x is still pretty poor. Having fast, JIT'ed code seems interesting in theory, but the way it's implemented in Guile 3.x is a giant headache: the separate byte compilation is extremely slow, and it is hard to manage (where should the .go files be stored/installed, how/when are they generated etc.). It also doesn't match our use case, because a lot of the code that we have comes from .ly files, so it cannot be precompiled. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Test reader speed of Guile 3.0.6
On Fri, Mar 12, 2021 at 12:39 PM Han-Wen Nienhuys wrote: > > there’s a Guile 3.0.6 release planned that includes a rewrite of the > > reader in Scheme. It has speed in the same order of magnitude as the > > previous reader but might have different performance characteristics. > > > > If I remember correctly, lilypond uses the reader a lot, so if you have > > a test-system with lilypond on Guile 3, could you test how running > > lilypond with the current Guile master from git affects lilypond? > > last time I looked, building GUILE 3 from source was truly glacial, > making this kind of thing annoying to check. > > You say "same order of magnitude". Do you have benchmarks so we know > what to expect? Also, as Harm describes, the build requires gperf, but you autoconf script doesn't check for it, as it should. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Test reader speed of Guile 3.0.6
On Wed, Mar 10, 2021 at 8:23 AM Dr. Arne Babenhauserheide wrote: > > Hi, > > there’s a Guile 3.0.6 release planned that includes a rewrite of the > reader in Scheme. It has speed in the same order of magnitude as the > previous reader but might have different performance characteristics. > > If I remember correctly, lilypond uses the reader a lot, so if you have > a test-system with lilypond on Guile 3, could you test how running > lilypond with the current Guile master from git affects lilypond? last time I looked, building GUILE 3 from source was truly glacial, making this kind of thing annoying to check. You say "same order of magnitude". Do you have benchmarks so we know what to expect? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Parsed object should be dead
On Sat, Feb 6, 2021 at 7:33 PM Dan Eble wrote: > Did something change in the past couple of weeks to make "Parsed object > should be dead" messages more likely? Now that I've turned from > documentation back to development, they seem to be getting in my way much > more than they used to. > > This stands out: > > commit d1be5ec60a058e363835bee60ece6fc61a67c5a9 > Author: Han-Wen Nienhuys > Date: Sun Jan 24 16:50:19 2021 +0100 > > Introduce 'debug-gc-lifetimes option > > The debug-gc-assert-parsed-dead feature has been documented, but it is > really an internal mechanism (it only has effect during garbage > collection, and has to be coupled with calls to ly:parsed-undead-list! > in order to do something useful. > > Instead, introduce the debug-gc-object-lifetimes option, which > controls the entire check (ie. the extra GC call, and the printing of > objects afterwards). For backward compatibility, this feature defaults > to #t. > > I wonder if we could disable this by default. As a user, I don't like > getting 672 lines of console output when I would have gotten 31 with this > option disabled. > I think we should default debug-gc-object-lifetimes to #f, and set it to #t for the regression test suite. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Scoping of Guile modules
On Sat, Feb 6, 2021 at 7:21 PM Jonas Hahnfeld wrote: > While trying to understand the current situation of LilyPond and its > Guile modules, I noticed that there is a certain split: > * lily.scm and all files pulled in by ly:load are referenced relative > to scm/, ie just the file name (without the extension after my changes > at https://gitlab.com/lilypond/lilypond/-/merge_requests/635 ). > * On the other hand, proper modules are prefixed with (scm), for > example (scm framework-ps) and (scm output-ps). > > This works right now, but requires that main_with_guile prepends both > the determined datadir and datadir + "/scm" to the %load-path. Similar > treatment would be necessary for %load-compiled-path for the compiled > bytecode. That's a bit ugly because it requires an scm/ directory to > hold the compiled .go files... > > I experimented with this locally and it seems possible to remove the > (scm) scope and reference all modules relative to scm/, ie have modules > (framework-ps), (output-ps) and so on. My question would be: Is there a > good reason *not* to do this? > As far as I could find out, this split goes back to the very first > module added in commit > > https://gitlab.com/lilypond/lilypond/-/commit/e11dc9a89c31b64615bcdcb8b536621ded30176b > from 2001. I'm adding Han-Wen and Jan, do you happen to remember if > that was an explicit choice or "just worked"? > I think this is a setup that stuck to the wall when we threw it, so there is no urgent reason to keep the naming precisely like this. However, other Scheme modules are grouped (eg, "(srfi srfi-1)" or "(ice-9 optargs)"). I think it would be good to follow this style and have a lilypond-specific prefix. Maybe "(lily framework-ps)" ? Are you trying to create a setup where the byte-compiled files are preinstalled? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Updating the CI image and bumping requirements
On Wed, Jan 27, 2021 at 8:12 PM Jonas Hahnfeld wrote: > with 2.23.0 out, I'd like to update the CI image. The current one is > based on Ubuntu 16.04 which will be EOL in April, a bit more than 2 > months from now. I think there are two logical replacements: > * Ubuntu 18.04, maintained until 2023-04 > * Debian 9 Stretch, maintained until 2022-06 > > Neither of them has Guile 1.8 packaged, so that'll have to be compiled > when building the image anyway, but I'd like to go with Ubuntu 18.04 > because it has Guile 2.2. In contrast, Debian 9 Stretch only comes with > guile-2.0 and I was surprised to find that LilyPond master doesn't even > compile with 2.0 at the moment [1]. > However, there's another consideration: Ubuntu 18.04 comes with Python > 3.6 and support for Python 3.5 (the current minimum & in Ubuntu 16.04) > is likely to break without proper testing. For that reason, I think we > should bump the requirement accordingly. The downside is that doing so > would effectively drop support for Debian 9 Stretch with Python 3.5. I > think that's ok given that Debian 10 Buster is the current version, but > wanted to check if that's a problem for anybody? > [If we agree on Ubuntu 18.04 as a kind of "minimum distribution", it > might make sense to bump some other requirements (such as Pango to get > rid of some compatibility code), but this can be done in due time.] What is the downside of sticking with current 16.04? Unless we make a conscious decision to drop support for GUILE 1.8, there is not much upside to moving to the newer versions. Dropping support for 1.8 has the potential to simplify a lot of GC related code, but IIRC, it's still a bit slower. > Cheers > Jonas > > > 1: Overlay_string_port relies on scm_c_make_port_with_encoding which > doesn't exist in Guile 2.0. I haven't looked if there's a replacement, > but I find it unlikely and think the time would be better spent on a > working version with Guile 2.2. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: probable guile-1.8.9 release
On Sat, Jan 16, 2021 at 2:52 PM Jonas Hahnfeld wrote: > > I can probably trim down the fix to be more backward compatible. > > > > Do you know of documentation that details what is and is not allowed > > in an ABI-compatible change? > > No, but let me know if you find one, would be good to have a reference > to a writeup from somebody who knows. > [..] > This means you must only change the body of functions and the size of > static variables (that are local per TU). Changing the meaning of only Thanks! How's this? https://github.com/hanwen/guile/commit/3bf5057232c35df155badfee9489081bdbbf4ecb -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: probable guile-1.8.9 release
On Sat, Jan 16, 2021 at 9:56 AM Jonas Hahnfeld via Discussions on LilyPond development wrote: > > Am Montag, dem 11.01.2021 um 11:06 +0100 schrieb Michael Käppler: > > Am 11.01.2021 um 10:58 schrieb Thomas Morley: > > > Hi, > > > > > > please have a look at > > > https://lists.gnu.org/archive/html/guile-user/2021-01/msg00026.html > > > > > > Cheers, > > >Harm > > > > > Could this probably include Han-Wen's recent fix regarding GC? > > My understanding is that the fix is ABI incompatible with earlier > releases of Guile 1.8; at least I get > $ LD_LIBRARY_PATH=/code/guile/install/lib/ ./out/bin/lilypond -dhelp > ./out/bin/lilypond: Symbol `scm_i_master_freelist2' has different size in > shared object, consider re-linking > ./out/bin/lilypond: Symbol `scm_i_master_freelist' has different size in > shared object, consider re-linking > > This is because scm_t_cell_type_statistics, the type of the two > symbols, changed size (got bigger), and this is usually a no-no for a > minor bugfix release. I'm not saying this makes sense (AFAICT these > symbols shouldn't be public), but it's the current situation and the > bugfix would require relinking all objects depending on the library. I can probably trim down the fix to be more backward compatible. Do you know of documentation that details what is and is not allowed in an ABI-compatible change? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Stale branches in the canonical repository
On Sun, Jan 10, 2021 at 9:24 PM Carl Sorensen wrote: > Rune's branch is left around as a tribute, I believe. Back in the day > (2.10 or so?) he completed a lot of significant work on Lilypond, then died > by suicide. > > I don't think the branch was actively intended as a tribute; it was just never cleaned up. We did dedicate a release to him ( https://lilypond.org/website/misc/announce-v2.12) > I believe that at this time, it would be appropriate to remove Rune' > branch. But I am only one person. > I think it's OK too. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Releasing 2.22.0
On Sun, Jan 3, 2021 at 12:06 PM Jonas Hahnfeld wrote: > There were no bug reports for 2.21.82 and no other Critical issue that > I'm aware of. While there are a few changes that could be backported (I > tried to label the MRs at GitLab), What is the label to look for? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Releasing 2.22.0
On Sun, Jan 3, 2021 at 12:06 PM Jonas Hahnfeld wrote: > Hi all and happy new year 2021 to everyone! happy new year to everyone, and I hope everyone is healthy and safe. > There were no bug reports for 2.21.82 and no other Critical issue that > [..] Sounds good to me Thanks for leading the way with the release! -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Would like to develop
On Thu, Dec 31, 2020 at 4:05 PM Juan Lucas Rey wrote: > Hello > > I am currently an Engineer working regularly with C++17 at Bloomberg, and > would like to be a contributor to lilypond. In particular I would like to > take as first objective to make it compatible with 64 bit compilation. > > Is this an objective that is desirable by the rest of the team? Was it > attempted before? I think LilyPond already works well on 64-bit architectures. (I'm using it on a x86_64 laptop currently). What makes you think it needs more work? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: output-distance is broken
On Wed, Dec 30, 2020 at 6:23 PM Dan Eble wrote: > On Dec 30, 2020, at 11:36, Dan Eble wrote: > > > > output-distance fails to detect significant differences (added text, > removed bar numbers) in this change: > > > > https://gitlab.com/lilypond/lilypond/-/merge_requests/582 > > > https://lilypond.gitlab.io/-/lilypond/-/jobs/938180873/artifacts/test-results/index.html > > Hm... there are no *.signature files for that case. > Looks like the known issue that \book (used for testing page breaking) doesn't generate .signature files. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Using 'libfaketime' for reproducible builds
On Mon, Dec 28, 2020 at 10:26 AM Jonas Hahnfeld wrote: > Am Sonntag, dem 27.12.2020 um 22:24 +0100 schrieb Werner LEMBERG: > > > Intercepting syscalls (or whatever the library does, I didn't > > > check) doesn't sound like the right approach outside of testing > > > reproducibility. > > > > Why? It's even less intrusive than the `SOURCE_DATE_EPOCH` solution. > > I definitely consider intercepting various syscalls by means of > LD_PRELOADing more intrusive than setting a single environment variable > that was invented for the purpose of setting timestamps. Just think of > a new shiny syscall that might add a new source of non-reproducibility. > I agree with Jonas. As a further argument, LD_PRELOAD is also dependent on the platform; I think it wouldn't work on OSX, for example. > > > I think that's a pity, but nothing we can change as a > > > "consumer" of library functions. > > > > Exactly. As long as we don't change LilyPond to produce PDFs by > > itself – which is a huge undertaking that I certainly won't start – > > I think we have no other choice than using something like > > 'libfaketime' or a patched gs version. I definitely prefer the > > former. > > What I wanted to say is that we cannot change the developers' minds to > support the environment variable. But we can (and IMHO should) use all > available interfaces if we care about reproducibility. I see at least > two more options: > Yes, +1 from me. > > 1) Strip non-determinism from the generated PDF. This is even mentioned > at https://reproducible-builds.org/docs/timestamps/ - before discussing > libfaketime which spends more than half of the paragraph mentioning > possible issues. > > 2) As we control the input PS code, we don't have to worry about the > operators that get the current time, draw a random number, etc. (as > long as we don't use them ourselves). Instead the bug linked above says > we just need to tell GS which CreationDate and ModDate to use (via > PDFmarks) and this should be straight-forward to fill with values > depending on SOURCE_DATE_EPOCH. > This probably leaves the UUIDs (is that the issue you mention above?) > which can be overridden using -sDocumentUUID and -sInstanceUUID. > Setting a constant time using libfaketime will result in the same UUID > for all generated PDFs, so it can't get worse; but I think it would be > desirable to do better than that and compute a "unique" ID based on the > input file, maybe as simple as the hash of the file path. It must be > considered that different values will prevent reuse of the GS API > instance, but I'd argue that a constant value should be fine in this > case. > the man for DocumentUUID says Note that Ghostscript has no assess to the host node ID due to a minimization of platform dependent modules. Therefore it uses an MD5 hash of the document contents for generating UUIDs. I wonder if we'd get reproducible documents if we provide only InstanceUUID -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Welcome to Travis CI!
Am Donnerstag, dem 17.12.2020 um 11:26 + schrieb Travis CI: > <http://travis-ci.com> = Welcome to Travis CI, GNU > LilyPond (unofficial)! = "GNU LilyPond (unofficial)" is the name of the Github organization for LilyPond, which has only 5 members. I don't know how this happened. Logging in requires giving Travis an absurd number of permissions. Let's just ignore it? On Fri, Dec 18, 2020 at 7:41 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > Hm, so I do use Travis at work, but I hope I didn't click anywhere I > shouldn't and this wasn't me. Does anybody have plans to use Travis? > -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Automatic 'make check' in CI
On Mon, Nov 23, 2020 at 9:11 PM Jonas Hahnfeld wrote: > Am Sonntag, den 22.11.2020, 22:49 +0100 schrieb David Kastrup: > > Jonas Hahnfeld writes: > > > > > Nope, that was a red herring: The reason is that the footnote creation > > > process in footnote-volta-spanner.ly messes with (point-stencil), > which > > > is used by skyline-point-extent.ly and also the BendSpanner. As far as > > > I understand the Stencil class, its objects are not immutable... > > > > > > One of the following two changes fixes the problem in my local test > > > scenario: > > > > > > [...] > > > > > > My gut feeling is that we should fix null-markup because nothing should > > > ever translate / rotate / modify the global point-stencil, thoughts? > > > > To me that seems crazy. Instead \footnote should not modify the stencil > > generated from its argument in place. If it needs something with > > different dimensions, it needs to create a new stencil, copying the > > stencil expression and changing the dimensions. > > Yes indeed, this seems to be the contract for stencils: Make a copy > whenever a Stencil is passed in from Scheme. I managed to find the code > path for Balloon grobs that violate this rule, here's a fix: > https://gitlab.com/lilypond/lilypond/-/merge_requests/522 > (Note that the test case shows that this was already broken before this > unstable series, at least 2.20.0 but probably older versions too). > thanks for tracking this down! point-stencil is from 2004, and the balloon code from thereabouts too. It's surprising that this took so long to turn up. > If possible, I'd like to merge this rather sooner than later so I can > finally post the MR to enable 'make check'.. > fine to skip the countdown for me. > > ly:stencil-set-extent! has been removed in 2.5.17. Extents are not > > intended to be mutable as far as I can discern, so if our C++ code does > > violate that assumption, it is that code we should fix instead of giving > > up on behavior guaranteed from the Scheme side of things. > > In fact, the C++ code does and many methods of the Stencil class modify > the object, but this is ok as long as we stick to the rule above. I'd > be happy to assert in some way that stencil representations coming from > Scheme are not altered, but I can't think of a good way to do this... > We'd have to make unsmob return a Stencil const *. Maybe that is appropriate for more simple smobs? -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Automatic 'make check' in CI
On Sun, Nov 22, 2020 at 1:55 PM Jonas Hahnfeld wrote: > Am Samstag, den 21.11.2020, 18:50 +0100 schrieb Jonas Hahnfeld: > > Am Samstag, den 21.11.2020, 09:46 +0100 schrieb Jonas Hahnfeld: > > > Am Freitag, den 20.11.2020, 17:55 -0500 schrieb Dan Eble: > > > > On Nov 20, 2020, at 12:22, Jonas Hahnfeld wrote: > > > > > > > Despite these shortcomings, I think it would make sense to > enable > > > > > > > this first implementation in lilypond/lilypond. What do you > think? > > > > > > > > > > > > > > > > > > > yes, +1 . I am especially happy that this will likely have no > extra > > > > > > overhead. > > > > > > > > > > No extra overhead per pipeline in MRs, but you get an extra > pipeline > > > > > for every merge to master. We'll see if this poses a serious > problem… > > > > > > > > Given the positive feedback this change has already received, can we > > > > accelerate its progress? I'm eager to have it in place because I've > > > > got a branch with new some regression tests ready. > > > > > > Sure, I can merge !517 later today and then work on the second MR to > > > actually enable 'make check' in the pipelines. > > > > Something's not working correctly here: > > > https://hahnjo.gitlab.io/-/lilypond/-/jobs/864846942/artifacts/test-results/index.html > > The test-baseline was generated by frog (using parallelism), check > > executed on a shared runner. I thought I had tested using a test- > > baseline with different number of cores... > > What I probably checked is various values for CPU_COUNT, including just > one core, and all works correctly AFAICT. What gives differences is not > setting CPU_COUNT / job-count at all. > > I successfully traced this back to compiling, for example, footnote- > volta-spanner.ly and skyline-point-extent.ly (in that order!) with an > added '\include "lilypond-book-preamble.ly"' (attached). If I run > $ ./out/bin/lilypond footnote-volta-spanner.ly skyline-point-extent.ly > the skyline-point-extent.pdf has some added whitespace that vanishes > when passing -djob-count=1 or 2 (starting with 3, it won't fork because > there are less input files than jobs). > > footnote-volta-spanner.ly was added by > commit c1f44faf958aafe7802f432a5ae0f4c74a8c0146 > Author: Han-Wen Nienhuys > Date: Tue Jun 16 08:59:35 2020 +0200 > > For footnotes on spanners, copy bounds from underlying spanner. > > This is likely more correct for spanners that run from non-musical > columns (eg. volta brackets) or are created retroactively > (eg. automatic beams). > > Demonstrate this by adding a regression test that puts a footnote on a > VoltaBracket. > > Without adjustments of Footnote_engraver, the test produces the > following: > > programming error: Spanner `FootnoteSpanner' is not fully contained > in parent spanner. Ignoring orphaned part > continuing, cross fingers > programming error: bounds of this piece aren't breakable. > continuing, cross fingers > > (https://gitlab.com/lilypond/lilypond/-/merge_requests/138). During > review, David had some concerns about Spanners that were dismissed on > the basis that there was no test case that showed wrong results (is > that really our standard for claiming that a change works correctly?!). > > I tried reverting that commit and all other differences also went away. > Does this mean the change is indeed flawed? > I'll try to have a look this week (likely Friday). It may be prudent to revert the change if 2.22 is imminent. Any ideas, comments? > Let's not reverse the arg list after fork, to avoid -djob-count confusing things more. I'd like to get rid of the fork call -that would also solve the problem with the options regtest. Let me try to have another look at !204. I am not too optimistic, but it seems worth trying another time. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Automatic 'make check' in CI
On Wed, Nov 18, 2020 at 9:25 PM Jonas Hahnfeld wrote: > Hi all, > > I'd like to present a first workable version of 'make check' for use in > our CI pipelines. I've pushed the necessary commits to my personal fork > and created two merge requests to demonstrate the results: > > 1. Run 'make check' for merge requests (no difference) > URL: https://gitlab.com/hahnjo/lilypond/-/merge_requests/5 > Job: https://gitlab.com/hahnjo/lilypond/-/jobs/858498690 > test-results: > > https://hahnjo.gitlab.io/-/lilypond/-/jobs/858498690/artifacts/test-results/index.html > > 2. Introduce difference in bar-lines.ly > URL: https://gitlab.com/hahnjo/lilypond/-/merge_requests/6 > Job: https://gitlab.com/hahnjo/lilypond/-/jobs/852618720 > test-results: > > https://hahnjo.gitlab.io/-/lilypond/-/jobs/852618720/artifacts/test-results/index.html > > This first workable implementation contains the minimum functionality: > It runs 'make test-baseline' for every push to the master branch and > replaces 'make test' in the pipelines for MRs with 'make check'. The > test-results are uploaded as artifacts and can be either downloaded as > zip archive or viewed directly (see above). > > There are a few known issues that I'm aware of: > - I needed to delete input/regression/option-help.ly because it logs > the options currently in use by lilypond, including the job-count which > varies when using our own runners with more than one core. > we could overwrite the job-count option in the option help, as it is irrelevant by the time you get to the file processing. > - Sometimes the test-results contain spurious diffs, for example here: > > https://hahnjo.gitlab.io/-/lilypond/-/jobs/858441670/artifacts/test-results/index.html > I can reproduce this locally with --enable-checking, but haven't > investigated further yet (there were a couple of other problems that I > needed to solve in order to get things working...). If somebody has an > idea for a fix, that would be great but I think these can be safely > ignored for now. > This could happen because of false-positives in the conservative garbage collection. It's not super-likely, but at the same time, it can't be ruled out. Does it always happen with the same files? I have never seen this, and the files are not doing anything out of the ordinary. IIRC, the dead-object detection can't be made to work anyway with GUILE 2.x, so this might be a good moment to scrap it. > There are a few more elaborate things that I'd like to work on in the > future. For example, GitLab can show a list of 'failing' tests which > can tell us at first glance if we need to look into the test-results. > I've prototyped this integration in the second MR, but it's very > misleading because the file extensions are missing and GitLab prints "0 > failed out of null" when there are no tests. The obvious solution is to > mark all existing tests as success, but this requires a bit more > thought to integrate into output-distance.py (or somewhere else). > I was going to suggest to use junit XML files, but it looks you already found this. Fantastic! > Despite these shortcomings, I think it would make sense to enable this > first implementation in lilypond/lilypond. What do you think? > yes, +1 . I am especially happy that this will likely have no extra overhead. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: LilyPond disabled on Wikimedia
urity audit. I’ve CC’d Tim Starling who performed the audit to this > thread, and he’s be in a better position to responsibly disclose problems. > > > > We hope to get LilyPond back on the Wikis, and that vulnerabilities get > fixed well for a safer LilyPond! > > > > Étienne > > > > Le 15 oct. 2020 à 19:05, Carl Sorensen a écrit : > > > > Unfortunately, there's not enough information on that thread to understand > what the issues are. > > I know that in the past there have been significant security concerns > which had a core concern related to Guile programming, since Guile is a > turing-complete language. > > I don't know how we can contribute until we are made aware of the > challenges here. > > Carl > > > On 10/15/20, 4:14 PM, "lilypond-devel on behalf of Daniel Benjamin Miller" > behalf of dbmil...@dbmiller.org> wrote: > > Not of direct relevance to us as end users, but can someone shed light > on this and/or resolve the concern of the Wikimedia people? In the > meantime Lilypond support has been disabled on Wikipedia. > https://phabricator.wikimedia.org/T257066 > > > > > > -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: plan for branching
On Tue, Oct 13, 2020 at 3:11 PM Jonas Hahnfeld wrote: > With the localization issues hopefully resolved and no objecting > comments to the last thread, I think we should go ahead and create > stable/2.22, with release candidates (2.21.8x) on the road to 2.22.0. > > After creating the branch, I will (unless somebody doesn't want me to > do this and volunteers to do the work; see also below): > - bump master to 2.23.0, in preparation for the next cycle; > - bump stable/2.22 to 2.21.80, in preparation for the first release > candidate; > - merge and / or forward the translation branch to stable/2.22. > > Thanks for taking this on! > - master is open again for feature development etc., eventually > released as 2.23.0 after 2.22.0 is done (note that it would be helpful > to avoid large-scale changes that complicate the following points); > I don't have plans for large scale changes (or any changes, actually) in the coming 1-2 months: I have been pretty busy at work last month, and I am moving apartments next week. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: strange replacement 100 → hundred in NR
On Wed, Sep 30, 2020 at 10:11 PM Han-Wen Nienhuys wrote:
>
>
> On Wed, Sep 30, 2020 at 8:05 PM Werner LEMBERG wrote:
>
>>
>> [git 9c3803fba4960b4afa63baeb50201a0cfa48f8f1]
>>
>> I've just built the whole documentation, and I get a strange string
>> replacement in Appendix A of the NR (from file
>> `markup-list-commands.texi`), see attached image.
>>
> As far as I remember, this didn't happen previously. Is it possible
>> that this code from `input.itely`
>>
>> \paper {
>> #(add-text-replacements!
>> '(("100" . "hundred")
>> ("dpi" . "dots per inch")))
>> }
>>
>> changes the state of lilypond while lilypond-book now processes more
>> files in one run?
>>
>
> Yes, very likely. See commit 9f16e2962ac71e6a30526012e314d55cec6b7e01.
>
> For some reason the check for replacement_cache_alist_key_ is not
> working as intended.
>
It looks like the C++ code is working as intended, but
#(define (add-text-replacements! alist)
(set! text-font-defaults
(assoc-set! text-font-defaults 'replacement-alist
(cdaar
(internal-add-text-replacements (list
text-font-defaults) alist)
modifies text-font-defaults in-place. It looks like the definition being
changed is the default paper setting, which is shared across files.
what was the last version in which this was seen working as expected?
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: strange replacement 100 → hundred in NR
On Wed, Sep 30, 2020 at 8:05 PM Werner LEMBERG wrote:
>
> [git 9c3803fba4960b4afa63baeb50201a0cfa48f8f1]
>
> I've just built the whole documentation, and I get a strange string
> replacement in Appendix A of the NR (from file
> `markup-list-commands.texi`), see attached image.
>
As far as I remember, this didn't happen previously. Is it possible
> that this code from `input.itely`
>
> \paper {
> #(add-text-replacements!
> '(("100" . "hundred")
> ("dpi" . "dots per inch")))
> }
>
> changes the state of lilypond while lilypond-book now processes more
> files in one run?
>
Yes, very likely. See commit 9f16e2962ac71e6a30526012e314d55cec6b7e01.
For some reason the check for replacement_cache_alist_key_ is not
working as intended.
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Run formatting tools
On Fri, Sep 25, 2020 at 9:36 AM Jonas Hahnfeld wrote: > Am Freitag, den 25.09.2020, 09:31 +0200 schrieb Werner LEMBERG: > > > What puzzles me a bit is the magnitude of changes from autopep8 - I > > > thought this tool was run in August? Maybe I'm missing something; > > > FWIW I used the invocation "autopep8 -ia --ignore=E402" as > > > documented in CG. > > > > > > Thoughts, objections? Should I exclude running autopep8? > > > > I was told from a Python professional developer that the best tool for > > formatting python code be `black`. Running > > > > black -l 78 ... > > > > gives indeed very nice formatting (not tested with lilypond stuff, > > though). > > Hm, I thought we agreed on using autopep8 more than two months ago? I'm > not very familiar with either tool but I'm against switching back and > forth multiple times, that'll just cause churn in the code. > IIRC I think the option used was -aa (2x -a = aggressive.) -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
